Jul 09 16:50:51.758778 ip-10-0-139-29 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 16:50:52.105373 ip-10-0-139-29 kubenswrapper[2563]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:50:52.105373 ip-10-0-139-29 kubenswrapper[2563]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 16:50:52.105373 ip-10-0-139-29 kubenswrapper[2563]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:50:52.105373 ip-10-0-139-29 kubenswrapper[2563]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 16:50:52.105373 ip-10-0-139-29 kubenswrapper[2563]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 16:50:52.107011 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.106919    2563 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 16:50:52.111094 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111058    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:52.111228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111217    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:52.111289 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111282    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:52.111334 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111312    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:52.111334 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111322    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:52.111334 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111329    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:52.111334 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111335    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111340    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111344    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111367    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111397    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111403    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111408    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111413    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111420    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111425    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111431    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111436    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111441    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111446    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111450    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111455    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:52.111498 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111467    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111613    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111620    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111624    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111626    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111630    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111632    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111635    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111637    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111640    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111642    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111645    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111648    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111651    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111654    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111656    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111659    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111661    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111664    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111667    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:52.111898 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111670    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111672    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111676    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111678    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111681    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111683    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111686    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111688    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111691    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111693    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111696    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111699    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111701    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111709    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111712    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111715    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111717    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111721    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111723    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111729    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:52.112377 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111733    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111738    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111741    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111743    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111746    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111749    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111752    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111754    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111759    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111763    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111766    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111769    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111772    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111775    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111778    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111780    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111783    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111785    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111788    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111790    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:52.112858 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111793    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111796    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111798    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111801    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111803    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111808    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111811    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111814    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111816    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111819    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111823    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111826    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111829    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111832    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111836    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111839    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.111842    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112630    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112639    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:52.113332 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112643    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112647    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112650    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112653    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112656    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112659    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112661    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112664    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112667    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112670    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112672    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112675    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112678    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112680    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112683    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112687    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112689    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112692    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112695    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112698    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:52.113802 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112701    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112705    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112708    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112711    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112714    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112717    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112720    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112723    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112725    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112728    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112730    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112733    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112735    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112738    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112740    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112743    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112746    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112748    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112751    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:52.114297 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112753    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112756    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112758    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112761    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112763    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112766    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112768    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112770    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112773    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112776    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112778    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112784    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112787    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112790    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112793    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112795    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112798    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112800    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112803    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112805    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:52.114762 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112808    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112812    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112815    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112818    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112821    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112823    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112826    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112828    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112831    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112834    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112836    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112839    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112841    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112844    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112847    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112849    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112852    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112854    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112857    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112860    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:52.115228 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112862    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112865    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112868    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112871    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112874    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112876    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112879    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112881    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112884    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112886    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112889    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112891    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112894    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112897    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112899    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112902    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112904    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.112907    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113301    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113306    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:52.115726 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113309    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113311    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113314    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113316    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113319    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113321    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113324    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113327    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113330    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113332    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113335    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113338    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113341    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113344    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113355    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113358    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113361    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113363    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113366    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113368    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:52.116189 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113371    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113373    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113376    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113378    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113382    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113385    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113388    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113390    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113393    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113395    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113398    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113401    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113403    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113405    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113408    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113410    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113413    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113416    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113418    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113420    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:52.116711 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113423    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113426    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113429    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113431    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113434    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113437    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113440    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113443    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113446    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113449    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113451    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113454    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113456    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113459    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113461    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113463    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113466    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113468    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113470    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113473    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:52.117187 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113477    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113495    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113498    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113501    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113504    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113507    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113510    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113512    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113516    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113519    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113521    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113524    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113526    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113529    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113532    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113535    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113538    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113540    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113543    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113546    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:52.117715 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113549    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113553    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113556    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113559    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113562    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113564    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113567    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113569    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113572    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113575    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113577    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113580    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113582    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113585    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113587    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113590    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.113592    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113663    2563 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113670    2563 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113676    2563 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 16:50:52.118203 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113680    2563 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113684    2563 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113687    2563 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113691    2563 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113695    2563 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113701    2563 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113705    2563 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113708    2563 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113712    2563 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113715    2563 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113718    2563 flags.go:64] FLAG: --cgroup-root=""
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113721    2563 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113723    2563 flags.go:64] FLAG: --client-ca-file=""
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113727    2563 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113730    2563 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113735    2563 flags.go:64] FLAG: --cluster-domain=""
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113738    2563 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113741    2563 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113744    2563 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113747    2563 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113751    2563 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113754    2563 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113757    2563 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 16:50:52.118732 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113760    2563 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113763    2563 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113766    2563 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113769    2563 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113772    2563 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113775    2563 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113779    2563 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113782    2563 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113785    2563 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113787    2563 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113790    2563 flags.go:64] FLAG: --enable-server="true"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113793    2563 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113798    2563 flags.go:64] FLAG: --event-burst="100"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113803    2563 flags.go:64] FLAG: --event-qps="50"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113807    2563 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113810    2563 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113813    2563 flags.go:64] FLAG: --eviction-hard=""
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113817    2563 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113820    2563 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113823    2563 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113827    2563 flags.go:64] FLAG: --eviction-soft=""
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113829    2563 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113833    2563 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113836    2563 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113839    2563 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 16:50:52.119309 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113842    2563 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113845    2563 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113847    2563 flags.go:64] FLAG: --feature-gates=""
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113851    2563 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113854    2563 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113857    2563 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113860    2563 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113863    2563 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113866    2563 flags.go:64] FLAG: --help="false"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113869    2563 flags.go:64] FLAG: --hostname-override="ip-10-0-139-29.ec2.internal"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113872    2563 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113874    2563 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113877    2563 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113881    2563 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113884    2563 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113887    2563 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113890    2563 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113892    2563 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113895    2563 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113898    2563 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113903    2563 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113907    2563 flags.go:64] FLAG: --kube-reserved=""
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113909    2563 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113912    2563 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 16:50:52.119989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113915    2563 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113918    2563 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113920    2563 flags.go:64] FLAG: --lock-file=""
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113923    2563 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113926    2563 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113929    2563 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113935    2563 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113938    2563 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113941    2563 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113944    2563 flags.go:64] FLAG: --logging-format="text"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113947    2563 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113950    2563 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113953    2563 flags.go:64] FLAG: --manifest-url=""
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113956    2563 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113960    2563 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113963    2563 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113967    2563 flags.go:64] FLAG: --max-pods="110"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113970    2563 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113973    2563 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113975    2563 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113978    2563 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113982    2563 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113984    2563 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113987    2563 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113994    2563 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 16:50:52.120587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.113997    2563 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114000    2563 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114003    2563 flags.go:64] FLAG: --pod-cidr=""
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114007    2563 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114014    2563 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114017    2563 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114020    2563 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114023    2563 flags.go:64] FLAG: --port="10250"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114026    2563 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114028    2563 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0c1fc6d8c42a2de44"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114039    2563 flags.go:64] FLAG: --qos-reserved=""
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114043    2563 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114046    2563 flags.go:64] FLAG: --register-node="true"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114049    2563 flags.go:64] FLAG: --register-with-taints=""
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114053    2563 flags.go:64] FLAG: --registry-burst="10"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114056    2563 flags.go:64] FLAG: --registry-qps="5"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114059    2563 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114062    2563 flags.go:64] FLAG: --reserved-memory=""
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114066    2563 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114069    2563 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114072    2563 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114075    2563 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114078    2563 flags.go:64] FLAG: --runonce="false"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114081    2563 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114084    2563 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 16:50:52.121220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114087    2563 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114090    2563 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114092    2563 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114095    2563 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114098    2563 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114101    2563 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114104    2563 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114107    2563 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114110    2563 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114113    2563 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114116    2563 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114122    2563 flags.go:64] FLAG: --system-cgroups=""
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114124    2563 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114130    2563 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114132    2563 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114135    2563 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114145    2563 flags.go:64] FLAG: --tls-min-version=""
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114148    2563 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114151    2563 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114154    2563 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114157    2563 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114160    2563 flags.go:64] FLAG: --v="2"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114163    2563 flags.go:64] FLAG: --version="false"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114168    2563 flags.go:64] FLAG: --vmodule=""
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114171    2563 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 16:50:52.121837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.114175    2563 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114288    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114291    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114295    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114298    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114300    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114303    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114306    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114309    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114312    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114314    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114317    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114320    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114322    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114325    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114327    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114330    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114332    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114337    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114340    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:52.122445 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114342    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114345    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114347    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114350    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114352    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114355    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114358    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114361    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114363    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114366    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114368    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114375    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114377    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114380    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114383    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114385    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114388    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114390    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114393    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114396    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:52.122983 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114398    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114401    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114403    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114405    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114408    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114410    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114413    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114415    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114418    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114420    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114428    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114430    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114433    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114435    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114439    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114443    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114446    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114448    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114451    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:52.123456 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114454    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114458    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114461    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114464    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114467    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114473    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114476    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114479    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114498    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114504    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114508    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114512    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114515    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114517    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114520    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114523    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114525    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114528    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114530    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:52.123927 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114533    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114535    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114538    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114541    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114546    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114549    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114552    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114555    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114557    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114560    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114562    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114565    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114567    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114570    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114572    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114575    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114577    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114580    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114582    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114588    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:52.124385 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114591    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.114593    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.115186    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.122162    2563 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.122180    2563 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.122928    2563 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.122936    2563 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.122997    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123002    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123005    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123009    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123012    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123015    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123017    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:52.124882 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123020    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123023    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123025    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123028    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123031    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123033    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123036    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123039    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123041    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123044    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123046    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123049    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123052    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123054    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123057    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123059    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123062    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123065    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123067    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123071    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:52.125225 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123073    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123076    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123080    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123083    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123086    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123088    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123091    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123094    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123096    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123099    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123102    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123104    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123107    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123110    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123112    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123115    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123117    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123120    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123122    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123125    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:52.125714 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123127    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123130    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123132    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123135    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123137    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123140    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123142    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123145    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123149    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123153    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123156    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123159    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123162    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123166    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123170    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123174    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123177    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123180    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123183    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123186    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:52.126196 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123189    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123191    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123194    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123198    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123200    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123203    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123206    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123208    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123211    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123213    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123217    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123221    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123224    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123227    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123229    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123232    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123235    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123237    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123239    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123242    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:52.126701 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123245    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123247    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123250    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123252    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123255    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123260    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123264    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123267    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123271    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123274    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123276    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123279    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.123284    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123416    2563 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 16:50:52.127168 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123422    2563 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123425    2563 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123428    2563 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123430    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123433    2563 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123435    2563 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123438    2563 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123441    2563 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123443    2563 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123446    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123448    2563 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123451    2563 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123454    2563 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123456    2563 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123459    2563 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123462    2563 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123465    2563 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123468    2563 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123470    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 16:50:52.127525 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123473    2563 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123475    2563 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123478    2563 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123495    2563 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123499    2563 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123503    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123506    2563 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123509    2563 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123512    2563 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123515    2563 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123518    2563 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123520    2563 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123523    2563 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123526    2563 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123529    2563 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123531    2563 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123535    2563 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123539    2563 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123542    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 16:50:52.127998 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123544    2563 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123546    2563 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123549    2563 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123551    2563 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123554    2563 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123557    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123559    2563 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123562    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123564    2563 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123567    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123570    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123573    2563 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123575    2563 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123578    2563 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123581    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123583    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123586    2563 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123589    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123591    2563 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 16:50:52.128460 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123594    2563 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123596    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123599    2563 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123602    2563 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123605    2563 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123607    2563 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123610    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123612    2563 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123615    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123617    2563 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123620    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123622    2563 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123625    2563 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123628    2563 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123630    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123632    2563 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123635    2563 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123638    2563 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123640    2563 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123642    2563 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 16:50:52.129023 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123645    2563 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123647    2563 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123650    2563 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123654    2563 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123658    2563 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123661    2563 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123664    2563 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123667    2563 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123670    2563 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123673    2563 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123676    2563 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123678    2563 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123681    2563 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123683    2563 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123686    2563 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123689    2563 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123692    2563 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123694    2563 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123698    2563 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123700    2563 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 16:50:52.129510 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:52.123703    2563 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 16:50:52.130067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.123708    2563 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 16:50:52.130067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.123825    2563 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 16:50:52.130067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.126177    2563 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 16:50:52.130067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.127494    2563 server.go:1019] "Starting client certificate rotation"
Jul 09 16:50:52.130067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.127601    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 16:50:52.130067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.127628    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 16:50:52.130067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.130024    2563 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 16:50:52.131889 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.131872    2563 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 16:50:52.142064 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.142043    2563 log.go:25] "Validated CRI v1 runtime API"
Jul 09 16:50:52.144174 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.144158    2563 log.go:25] "Validated CRI v1 image API"
Jul 09 16:50:52.144992 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.144974    2563 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 16:50:52.147137 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.147117    2563 fs.go:135] Filesystem UUIDs: map[38ce0fb9-4c64-482c-b247-3f031f1e4cc1:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 9f9110fb-7b86-4bc2-afdf-1c93fadb500d:/dev/nvme0n1p4]
Jul 09 16:50:52.147204 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.147137    2563 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 16:50:52.153079 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.152968    2563 manager.go:216] Machine: {Timestamp:2026-07-09 16:50:52.151832756 +0000 UTC m=+0.292941303 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102915 MemoryCapacity:33164492800 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec27099c43b7e6f5a0ad619fef071ff3 SystemUUID:ec27099c-43b7-e6f5-a0ad-619fef071ff3 BootID:0fce6a55-043a-49f9-8d7b-a152231ff7c3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582246400 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:27:c6:57:ca:31 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:27:c6:57:ca:31 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:be:2e:35:7d:73:00 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164492800 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 16:50:52.153079 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.153078    2563 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 16:50:52.153185 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.153160    2563 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 16:50:52.155850 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.155821    2563 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 16:50:52.156007 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.155852    2563 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-139-29.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 16:50:52.156550 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.156541    2563 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 16:50:52.156583 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.156554    2563 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 16:50:52.156583 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.156566    2563 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 16:50:52.157152 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.157142    2563 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 16:50:52.157362 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.157346    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 16:50:52.158507 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.158496    2563 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 16:50:52.162314 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162301    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 16:50:52.162428 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162420    2563 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 16:50:52.162464 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162448    2563 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 16:50:52.162464 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162457    2563 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 16:50:52.162540 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162476    2563 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 16:50:52.162540 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162512    2563 kubelet.go:398] "Adding apiserver pod source"
Jul 09 16:50:52.162540 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162521    2563 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 16:50:52.162904 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162892    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 16:50:52.162957 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.162911    2563 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 16:50:52.165236 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.165216    2563 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 16:50:52.167197 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167184    2563 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 16:50:52.167236 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167215    2563 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 16:50:52.167370 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167361    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 16:50:52.167399 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167376    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 16:50:52.167399 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167382    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 16:50:52.167399 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167388    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 16:50:52.167497 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167409    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 16:50:52.167497 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167419    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 16:50:52.167868 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167858    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 16:50:52.167895 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167868    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 16:50:52.167895 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167875    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 16:50:52.167895 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167881    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 16:50:52.167895 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167892    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 16:50:52.167998 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.167902    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 16:50:52.168548 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.168536    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 16:50:52.168548 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.168548    2563 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 16:50:52.172042 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.172019    2563 server.go:1268] "Started kubelet"
Jul 09 16:50:52.172133 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.172070    2563 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 16:50:52.172698 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.172643    2563 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 16:50:52.172813 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.172722    2563 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 16:50:52.173821 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.173802    2563 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-139-29.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 16:50:52.173923 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:52.173805    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 16:50:52.173923 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:52.173881    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-139-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 16:50:52.174079 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.174061    2563 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 16:50:52.174292 ip-10-0-139-29 systemd[1]: Started Kubernetes Kubelet.
Jul 09 16:50:52.177549 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:52.177526    2563 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 16:50:52.177642 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.177550    2563 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 16:50:52.177986 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.177970    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5smw7"
Jul 09 16:50:52.180643 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:52.179878    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-29.ec2.internal.18c0ad2ae4779b70  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-29.ec2.internal,UID:ip-10-0-139-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-139-29.ec2.internal,},FirstTimestamp:2026-07-09 16:50:52.17199192 +0000 UTC m=+0.313100469,LastTimestamp:2026-07-09 16:50:52.17199192 +0000 UTC m=+0.313100469,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-29.ec2.internal,}"
Jul 09 16:50:52.180787 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.180773    2563 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 16:50:52.180787 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.180780    2563 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 16:50:52.180978 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:52.180902    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:52.181469 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181447    2563 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 16:50:52.181581 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181450    2563 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 16:50:52.181581 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181506    2563 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 16:50:52.181717 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181581    2563 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 16:50:52.181717 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181590    2563 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 16:50:52.181915 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181897    2563 factory.go:153] Registering CRI-O factory
Jul 09 16:50:52.181915 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181918    2563 factory.go:223] Registration of the crio container factory successfully
Jul 09 16:50:52.182056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181969    2563 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 16:50:52.182056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181978    2563 factory.go:55] Registering systemd factory
Jul 09 16:50:52.182056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.181985    2563 factory.go:223] Registration of the systemd container factory successfully
Jul 09 16:50:52.182056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.182007    2563 factory.go:103] Registering Raw factory
Jul 09 16:50:52.182056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.182022    2563 manager.go:1196] Started watching for new ooms in manager
Jul 09 16:50:52.182573 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.182557    2563 manager.go:349] Starting recovery of all containers
Jul 09 16:50:52.184834 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:52.184805    2563 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 16:50:52.184931 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:52.184851    2563 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-139-29.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 16:50:53.247571 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:52.186020    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-5smw7"
Jul 09 16:50:53.252880 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.252729    2563 manager.go:354] Recovery completed
Jul 09 16:50:53.257356 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.257340    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:53.259137 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.259120    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:53.259230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.259148    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:53.259230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.259158    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:53.259699 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.259685    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:53.259757 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.259709    2563 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 16:50:53.259757 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.259720    2563 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 16:50:53.259757 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.259739    2563 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 16:50:53.260375 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.260358    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:53.260440 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.260384    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:53.260440 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.260394    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:53.261261 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.261203    2563 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-139-29.ec2.internal.18c0ad2b25441b77  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-139-29.ec2.internal,UID:ip-10-0-139-29.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-139-29.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-139-29.ec2.internal,},FirstTimestamp:2026-07-09 16:50:53.259135863 +0000 UTC m=+0.339076826,LastTimestamp:2026-07-09 16:50:53.259135863 +0000 UTC m=+0.339076826,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-139-29.ec2.internal,}"
Jul 09 16:50:53.261835 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.261823    2563 policy_none.go:49] "None policy: Start"
Jul 09 16:50:53.261875 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.261840    2563 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 16:50:53.261875 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.261851    2563 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 16:50:53.263779 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.263766    2563 policy_none.go:47] "Start"
Jul 09 16:50:53.303286 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.303261    2563 manager.go:337] "Starting Device Plugin manager"
Jul 09 16:50:53.303405 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.303300    2563 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 16:50:53.303405 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.303311    2563 server.go:87] "Starting device plugin registration server"
Jul 09 16:50:53.303628 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.303606    2563 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 16:50:53.303799 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.303623    2563 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 16:50:53.303799 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.303747    2563 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 16:50:53.303902 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.303825    2563 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 16:50:53.303902 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.303834    2563 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 16:50:53.304379 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.304349    2563 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 16:50:53.304436 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.304407    2563 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:53.345433 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.345402    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 16:50:53.352874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.346609    2563 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 16:50:53.352874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.346672    2563 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 16:50:53.352874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.346704    2563 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 16:50:53.352874 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.346746    2563 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 16:50:53.352874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.348919    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:50:53.404640 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.404606    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:53.405630 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.405612    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:53.405737 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.405644    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:53.405737 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.405670    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:53.405737 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.405694    2563 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.411936 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.411923    2563 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.411993 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.411944    2563 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-139-29.ec2.internal\": node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:53.426586 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.426530    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:53.447088 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.447063    2563 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"]
Jul 09 16:50:53.447160 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.447123    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:53.448467 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.448451    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:53.448531 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.448481    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:53.448531 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.448496    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:53.450909 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.450894    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:53.450987 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.450974    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.451021 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.451001    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:53.451648 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.451623    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:53.451648 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.451663    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:53.451781 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.451674    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:53.452426 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.452412    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:53.452487 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.452440    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:53.452487 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.452450    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:53.454008 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.453992    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.454085 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.454018    2563 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 16:50:53.454733 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.454714    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 16:50:53.454836 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.454752    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 16:50:53.454836 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.454770    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeHasSufficientPID"
Jul 09 16:50:53.488183 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.488162    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-29.ec2.internal\" not found" node="ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.492541 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.492525    2563 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-139-29.ec2.internal\" not found" node="ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.526773 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.526754    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:53.627122 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.627093    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:53.643439 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.643413    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc2d5eb401e4b3f52c292a48eec32885-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"fc2d5eb401e4b3f52c292a48eec32885\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.643491 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.643453    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43807d716b03976d3a2bd26c692f140f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"43807d716b03976d3a2bd26c692f140f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.643491 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.643477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43807d716b03976d3a2bd26c692f140f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"43807d716b03976d3a2bd26c692f140f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.727841 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.727771    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:53.744062 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.744043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43807d716b03976d3a2bd26c692f140f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"43807d716b03976d3a2bd26c692f140f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.744109 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.744068    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43807d716b03976d3a2bd26c692f140f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"43807d716b03976d3a2bd26c692f140f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.744109 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.744089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc2d5eb401e4b3f52c292a48eec32885-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"fc2d5eb401e4b3f52c292a48eec32885\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.744167 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.744140    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/43807d716b03976d3a2bd26c692f140f-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"43807d716b03976d3a2bd26c692f140f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.744167 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.744148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/43807d716b03976d3a2bd26c692f140f-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal\" (UID: \"43807d716b03976d3a2bd26c692f140f\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.744228 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.744205    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fc2d5eb401e4b3f52c292a48eec32885-config\") pod \"kube-apiserver-proxy-ip-10-0-139-29.ec2.internal\" (UID: \"fc2d5eb401e4b3f52c292a48eec32885\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.790237 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.790213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.796076 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:53.795919    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:53.828774 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.828747    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:53.929259 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:53.929214    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:54.029842 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:54.029759    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:54.130267 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:54.130236    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:54.187804 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.187777    2563 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 16:50:54.188316 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.187939    2563 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 16:50:54.231328 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:54.231300    2563 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-139-29.ec2.internal\" not found"
Jul 09 16:50:54.252011 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.251976    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 16:45:53 +0000 UTC" deadline="2028-03-17 23:10:30.672746748 +0000 UTC"
Jul 09 16:50:54.252011 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.252004    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14814h19m36.42074607s"
Jul 09 16:50:54.294843 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.294771    2563 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:50:54.317854 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.317830    2563 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:50:54.333193 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.333173    2563 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:50:54.342197 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.342177    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:54.347883 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:54.347839    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43807d716b03976d3a2bd26c692f140f.slice/crio-c3faafcda60ebf40a8dfcd6e717a0d5b26933003ffbff05e8f88633631830908 WatchSource:0}: Error finding container c3faafcda60ebf40a8dfcd6e717a0d5b26933003ffbff05e8f88633631830908: Status 404 returned error can't find the container with id c3faafcda60ebf40a8dfcd6e717a0d5b26933003ffbff05e8f88633631830908
Jul 09 16:50:54.348395 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:54.348375    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc2d5eb401e4b3f52c292a48eec32885.slice/crio-d62e718a3247ed4032142aed0303e3c203a43859122534f70305b098617d65a3 WatchSource:0}: Error finding container d62e718a3247ed4032142aed0303e3c203a43859122534f70305b098617d65a3: Status 404 returned error can't find the container with id d62e718a3247ed4032142aed0303e3c203a43859122534f70305b098617d65a3
Jul 09 16:50:54.351568 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.351550    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 16:50:54.352395 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.352373    2563 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal"
Jul 09 16:50:54.353055 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.353038    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:50:54.361475 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:54.361458    2563 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 16:50:55.224205 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.224167    2563 apiserver.go:52] "Watching apiserver"
Jul 09 16:50:55.232441 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.232415    2563 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 16:50:55.233119 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.232875    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-operator/iptables-alerter-f8c9g","kube-system/global-pull-secret-syncer-cf425","openshift-cluster-node-tuning-operator/tuned-z2hh6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal","openshift-multus/multus-nz2rr","openshift-multus/network-metrics-daemon-qh9z6","openshift-ovn-kubernetes/ovnkube-node-5pxqp","kube-system/konnectivity-agent-6ccpn","kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws","openshift-dns/node-resolver-qkg46","openshift-image-registry/node-ca-tgdbb","openshift-multus/multus-additional-cni-plugins-2stnr","openshift-network-diagnostics/network-check-target-fvr59"]
Jul 09 16:50:55.235471 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.235450    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.237756 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.237724    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.237942 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.237923    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 16:50:55.238194 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.238024    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-m52f8\""
Jul 09 16:50:55.238194 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.238149    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 16:50:55.238194 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.238162    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 16:50:55.240167 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.239900    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.240167 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.239941    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 16:50:55.240167 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.239941    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-ctw2t\""
Jul 09 16:50:55.240465 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.240445    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 16:50:55.240553 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.240497    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 16:50:55.240937 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.240831    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 16:50:55.240937 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.240861    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 16:50:55.242091 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.242065    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-std9r\""
Jul 09 16:50:55.242210 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.242159    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:50:55.242210 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.242193    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 16:50:55.242421 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.242401    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:55.242532 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.242477    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:50:55.243032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.243014    2563 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 16:50:55.244728 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.244712    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.247053 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.247033    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-4wjld\""
Jul 09 16:50:55.247233 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.247156    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:55.247303 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.247236    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:50:55.247470 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.247444    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 16:50:55.249500 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.249447    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.251742 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251702    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49vjg\" (UniqueName: \"kubernetes.io/projected/57af9d2b-1213-446a-9d46-f43546d8603e-kube-api-access-49vjg\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.251840 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5wzs\" (UniqueName: \"kubernetes.io/projected/628ba919-14e4-4c4b-928c-f3518abff7e1-kube-api-access-d5wzs\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.251840 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251788    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-sys\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.251840 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251811    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-tuned\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.251840 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251836    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251863    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-cni-bin\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-conf-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251907    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-socket-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251931    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-sys-fs\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251952    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-cnibin\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.251974    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-kubernetes\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252011    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-cnibin\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.252032 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-kubelet\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252055    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-hostroot\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252077    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-daemon-config\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252100    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252111    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-p9d9q\""
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252123    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252149    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-var-lib-kubelet\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252194    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9stvx\" (UniqueName: \"kubernetes.io/projected/4ae3ca7c-05a4-4e68-b815-426cc000fa72-kube-api-access-9stvx\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252221    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-registration-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252257    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252283    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysconfig\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252314    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252358    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysctl-d\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252392    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-lib-modules\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252408    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.252435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-system-cni-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252458    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-os-release\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252527    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252553    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-system-cni-dir\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252590    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysctl-conf\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252617    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-socket-dir-parent\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252642    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-netns\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252698    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-cni-multus\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252727    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-os-release\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252778    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-systemd\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252810    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-host\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252849    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-cni-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252890    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a58a8df2-224f-4576-9d55-3fc4486f00f0-cni-binary-copy\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252939    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-device-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.253143 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252968    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-modprobe-d\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.254034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.252991    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-k8s-cni-cncf-io\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.254034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.253033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-etc-kubernetes\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.254034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.253058    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-run\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.254034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.253080    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ae3ca7c-05a4-4e68-b815-426cc000fa72-tmp\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.254034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.253103    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-multus-certs\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.254034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.253226    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b29rn\" (UniqueName: \"kubernetes.io/projected/a58a8df2-224f-4576-9d55-3fc4486f00f0-kube-api-access-b29rn\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.254736 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.254718    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 16:50:55.254828 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.254813    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mj768\""
Jul 09 16:50:55.255594 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.255428    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 16:50:55.255594 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.255548    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.255777 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.255612    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:50:55.256017 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.255998    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 16:50:55.256105 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.256018    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 16:50:55.256105 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.256067    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 16:50:55.257056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.256761    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 16:50:55.257364 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.257342    2563 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 16:50:55.259917 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.258296    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.261394 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.260933    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:50:55.261394 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.261280    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-bhc62\""
Jul 09 16:50:55.261394 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.261286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.261394 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.261308    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 16:50:55.261610 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.261497    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 16:50:55.263379 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.263354    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.263487 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.263469    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-v4lql\""
Jul 09 16:50:55.263606 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.263584    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 16:50:55.263729 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.263677    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 16:50:55.265510 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.265488    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 16:50:55.265789 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.265731    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-zgc9k\""
Jul 09 16:50:55.266155 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.265845    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 16:50:55.266155 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.265731    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 16:50:55.275646 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.275584    2563 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-2645k"
Jul 09 16:50:55.282343 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.282298    2563 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-2645k"
Jul 09 16:50:55.344114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.344080    2563 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 16:50:55.351519 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.351474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"43807d716b03976d3a2bd26c692f140f","Type":"ContainerStarted","Data":"c3faafcda60ebf40a8dfcd6e717a0d5b26933003ffbff05e8f88633631830908"}
Jul 09 16:50:55.353441 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353379    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" event={"ID":"fc2d5eb401e4b3f52c292a48eec32885","Type":"ContainerStarted","Data":"d62e718a3247ed4032142aed0303e3c203a43859122534f70305b098617d65a3"}
Jul 09 16:50:55.353441 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353407    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-hostroot\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.353675 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353441    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-ovn\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.353675 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.353675 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-hostroot\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.353675 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353573    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.353675 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353606    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-systemd\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.353675 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-cni-bin\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.353675 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d22hj\" (UniqueName: \"kubernetes.io/projected/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-kube-api-access-d22hj\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353720    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-cni-multus\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353751    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-registration-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353776    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353799    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysconfig\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353820    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-cni-multus\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353824    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-lib-modules\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353846    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-registration-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353902    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysconfig\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353930    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-system-cni-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353958    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-lib-modules\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.353962    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.353996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354000    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-system-cni-dir\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354025    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysctl-conf\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354048    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-netns\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354055    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-etc-selinux\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354064    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-system-cni-dir\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354097    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354108    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354120    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-netns\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354124    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2zqs\" (UniqueName: \"kubernetes.io/projected/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-kube-api-access-w2zqs\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-run-netns\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354185    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysctl-conf\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354199    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354238    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-cni-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354251    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-cni-binary-copy\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd66d1-cdef-4ed1-b094-5d664f273801-tmp-dir\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354277    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/628ba919-14e4-4c4b-928c-f3518abff7e1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354289    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-var-lib-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.354556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-cni-netd\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354319    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-cni-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-env-overrides\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354388    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354402    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354421    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-modprobe-d\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354425    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-system-cni-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354445    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-k8s-cni-cncf-io\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354477    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-log-socket\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354502    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354511    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-k8s-cni-cncf-io\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354519    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae1eeec8-a184-4829-b055-c7715ca287ff-kubelet-config\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354556    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-run\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354577    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-modprobe-d\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354587    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b29rn\" (UniqueName: \"kubernetes.io/projected/a58a8df2-224f-4576-9d55-3fc4486f00f0-kube-api-access-b29rn\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354613    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-etc-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-run\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49vjg\" (UniqueName: \"kubernetes.io/projected/57af9d2b-1213-446a-9d46-f43546d8603e-kube-api-access-49vjg\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354699    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5wzs\" (UniqueName: \"kubernetes.io/projected/628ba919-14e4-4c4b-928c-f3518abff7e1-kube-api-access-d5wzs\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354723    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-cni-bin\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354752    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-conf-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-conf-dir\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354792    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-cni-bin\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354835    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd4l8\" (UniqueName: \"kubernetes.io/projected/dc814b9e-7680-4729-9858-c192c4de754c-kube-api-access-rd4l8\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354860    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovnkube-config\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354895    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-sys-fs\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354919    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-kubernetes\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354949    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-daemon-config\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.354972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae1eeec8-a184-4829-b055-c7715ca287ff-dbus\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355005    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-sys-fs\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355023    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-var-lib-kubelet\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355045    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-kubernetes\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355092    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9stvx\" (UniqueName: \"kubernetes.io/projected/4ae3ca7c-05a4-4e68-b815-426cc000fa72-kube-api-access-9stvx\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355119    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-serviceca\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.355960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355129    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-var-lib-kubelet\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355168    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20ef3d43-c5db-49ef-8141-5cfbd35f397a-agent-certs\") pod \"konnectivity-agent-6ccpn\" (UID: \"20ef3d43-c5db-49ef-8141-5cfbd35f397a\") " pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4sxs\" (UniqueName: \"kubernetes.io/projected/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-kube-api-access-m4sxs\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355244    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysctl-d\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-os-release\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355304    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355339    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355399    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-socket-dir-parent\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355413    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-sysctl-d\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355435    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-host-slash\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355445    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-os-release\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355464    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-kubelet\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355465    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355466    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-socket-dir-parent\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355492    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-slash\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355504    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/a58a8df2-224f-4576-9d55-3fc4486f00f0-multus-daemon-config\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.356726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355520    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-os-release\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355579    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-os-release\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-systemd\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355619    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-host\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355637    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a58a8df2-224f-4576-9d55-3fc4486f00f0-cni-binary-copy\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355680    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-iptables-alerter-script\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-systemd\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355697    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-device-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355717    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-etc-kubernetes\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355740    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39cd66d1-cdef-4ed1-b094-5d664f273801-hosts-file\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355762    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mblls\" (UniqueName: \"kubernetes.io/projected/39cd66d1-cdef-4ed1-b094-5d664f273801-kube-api-access-mblls\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355777    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-host\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355790    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-device-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-systemd-units\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355832    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovn-node-metrics-cert\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-etc-kubernetes\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355889    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-host\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.357322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ae3ca7c-05a4-4e68-b815-426cc000fa72-tmp\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355934    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-multus-certs\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-node-log\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.355981    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovnkube-script-lib\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356003    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-sys\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356028    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-tuned\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356052    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356076    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20ef3d43-c5db-49ef-8141-5cfbd35f397a-konnectivity-ca\") pod \"konnectivity-agent-6ccpn\" (UID: \"20ef3d43-c5db-49ef-8141-5cfbd35f397a\") " pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356098    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-socket-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356113    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/a58a8df2-224f-4576-9d55-3fc4486f00f0-cni-binary-copy\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356167    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-cnibin\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356181    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/4ae3ca7c-05a4-4e68-b815-426cc000fa72-sys\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356191    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-cnibin\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356200    2563 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356223    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-run-multus-certs\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.356216    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-kubelet\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.357048    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-host-var-lib-kubelet\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.357875    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/57af9d2b-1213-446a-9d46-f43546d8603e-socket-dir\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.358047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.357927    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/628ba919-14e4-4c4b-928c-f3518abff7e1-cnibin\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.358643 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.358143    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/a58a8df2-224f-4576-9d55-3fc4486f00f0-cnibin\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.360639 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.360602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/4ae3ca7c-05a4-4e68-b815-426cc000fa72-tmp\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.362372 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.362328    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/4ae3ca7c-05a4-4e68-b815-426cc000fa72-etc-tuned\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.364766 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.364727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5wzs\" (UniqueName: \"kubernetes.io/projected/628ba919-14e4-4c4b-928c-f3518abff7e1-kube-api-access-d5wzs\") pod \"multus-additional-cni-plugins-2stnr\" (UID: \"628ba919-14e4-4c4b-928c-f3518abff7e1\") " pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.364869 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.364848    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9stvx\" (UniqueName: \"kubernetes.io/projected/4ae3ca7c-05a4-4e68-b815-426cc000fa72-kube-api-access-9stvx\") pod \"tuned-z2hh6\" (UID: \"4ae3ca7c-05a4-4e68-b815-426cc000fa72\") " pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.364935 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.364853    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:50:55.364935 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.364905    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:50:55.364935 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.364920    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m7rsh for pod openshift-network-diagnostics/network-check-target-fvr59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:55.365072 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.364991    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh podName:fb9c8550-2a74-4e96-8a11-42c778b6d711 nodeName:}" failed. No retries permitted until 2026-07-09 16:50:55.864971428 +0000 UTC m=+2.944912379 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-m7rsh" (UniqueName: "kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh") pod "network-check-target-fvr59" (UID: "fb9c8550-2a74-4e96-8a11-42c778b6d711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:55.365170 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.365150    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b29rn\" (UniqueName: \"kubernetes.io/projected/a58a8df2-224f-4576-9d55-3fc4486f00f0-kube-api-access-b29rn\") pod \"multus-nz2rr\" (UID: \"a58a8df2-224f-4576-9d55-3fc4486f00f0\") " pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.365440 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.365408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-49vjg\" (UniqueName: \"kubernetes.io/projected/57af9d2b-1213-446a-9d46-f43546d8603e-kube-api-access-49vjg\") pod \"aws-ebs-csi-driver-node-qgcws\" (UID: \"57af9d2b-1213-446a-9d46-f43546d8603e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.457635 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39cd66d1-cdef-4ed1-b094-5d664f273801-hosts-file\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.457635 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457638    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mblls\" (UniqueName: \"kubernetes.io/projected/39cd66d1-cdef-4ed1-b094-5d664f273801-kube-api-access-mblls\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457675    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-host\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457695    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-systemd-units\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457716    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovn-node-metrics-cert\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457738    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-node-log\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/39cd66d1-cdef-4ed1-b094-5d664f273801-hosts-file\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovnkube-script-lib\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457794    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20ef3d43-c5db-49ef-8141-5cfbd35f397a-konnectivity-ca\") pod \"konnectivity-agent-6ccpn\" (UID: \"20ef3d43-c5db-49ef-8141-5cfbd35f397a\") " pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457796    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-systemd-units\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457820    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-ovn\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457829    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-node-log\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457845    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-systemd\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.457862 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457868    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-cni-bin\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457881    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-ovn\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d22hj\" (UniqueName: \"kubernetes.io/projected/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-kube-api-access-d22hj\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457937    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-cni-bin\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457941    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-systemd\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457940    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-host\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457972    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2zqs\" (UniqueName: \"kubernetes.io/projected/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-kube-api-access-w2zqs\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.457998    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-run-netns\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458022    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd66d1-cdef-4ed1-b094-5d664f273801-tmp-dir\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.458040    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458077    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-var-lib-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.458117    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs podName:dc814b9e-7680-4729-9858-c192c4de754c nodeName:}" failed. No retries permitted until 2026-07-09 16:50:55.958098307 +0000 UTC m=+3.038039268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs") pod "network-metrics-daemon-qh9z6" (UID: "dc814b9e-7680-4729-9858-c192c4de754c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458114    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-run-netns\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458046    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-var-lib-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458155    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-cni-netd\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458179    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-env-overrides\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.458455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458211    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-log-socket\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458267    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458291    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae1eeec8-a184-4829-b055-c7715ca287ff-kubelet-config\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458301    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-cni-netd\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458319    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-etc-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458352    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-log-socket\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd4l8\" (UniqueName: \"kubernetes.io/projected/dc814b9e-7680-4729-9858-c192c4de754c-kube-api-access-rd4l8\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458386    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/39cd66d1-cdef-4ed1-b094-5d664f273801-tmp-dir\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovnkube-config\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458424    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovnkube-script-lib\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.458466    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458489    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/ae1eeec8-a184-4829-b055-c7715ca287ff-kubelet-config\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.458516    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret podName:ae1eeec8-a184-4829-b055-c7715ca287ff nodeName:}" failed. No retries permitted until 2026-07-09 16:50:55.95849797 +0000 UTC m=+3.038438934 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret") pod "global-pull-secret-syncer-cf425" (UID: "ae1eeec8-a184-4829-b055-c7715ca287ff") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458530    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-run-ovn-kubernetes\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458539    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae1eeec8-a184-4829-b055-c7715ca287ff-dbus\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458568    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-etc-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459258 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458581    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-env-overrides\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458570    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-serviceca\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458618    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20ef3d43-c5db-49ef-8141-5cfbd35f397a-agent-certs\") pod \"konnectivity-agent-6ccpn\" (UID: \"20ef3d43-c5db-49ef-8141-5cfbd35f397a\") " pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458696    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4sxs\" (UniqueName: \"kubernetes.io/projected/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-kube-api-access-m4sxs\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458726    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458756    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458791    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-host-slash\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458814    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-kubelet\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458838    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-slash\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458867    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-iptables-alerter-script\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458918    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovnkube-config\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458954    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/20ef3d43-c5db-49ef-8141-5cfbd35f397a-konnectivity-ca\") pod \"konnectivity-agent-6ccpn\" (UID: \"20ef3d43-c5db-49ef-8141-5cfbd35f397a\") " pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/ae1eeec8-a184-4829-b055-c7715ca287ff-dbus\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458977    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-run-openvswitch\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.458968    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-serviceca\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.459016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-host-slash\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.459874 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.459059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-kubelet\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.460434 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.459100    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-host-slash\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.460434 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.459389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-iptables-alerter-script\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.460728 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.460706    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-ovn-node-metrics-cert\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.461147 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.461126    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/20ef3d43-c5db-49ef-8141-5cfbd35f397a-agent-certs\") pod \"konnectivity-agent-6ccpn\" (UID: \"20ef3d43-c5db-49ef-8141-5cfbd35f397a\") " pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.468670 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.468634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd4l8\" (UniqueName: \"kubernetes.io/projected/dc814b9e-7680-4729-9858-c192c4de754c-kube-api-access-rd4l8\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:55.468912 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.468891    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d22hj\" (UniqueName: \"kubernetes.io/projected/75f17ac6-afdd-4ff0-8345-d11aff9e54b6-kube-api-access-d22hj\") pod \"ovnkube-node-5pxqp\" (UID: \"75f17ac6-afdd-4ff0-8345-d11aff9e54b6\") " pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.469055 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.469033    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mblls\" (UniqueName: \"kubernetes.io/projected/39cd66d1-cdef-4ed1-b094-5d664f273801-kube-api-access-mblls\") pod \"node-resolver-qkg46\" (UID: \"39cd66d1-cdef-4ed1-b094-5d664f273801\") " pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.469312 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.469293    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2zqs\" (UniqueName: \"kubernetes.io/projected/f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f-kube-api-access-w2zqs\") pod \"iptables-alerter-f8c9g\" (UID: \"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f\") " pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.469397 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.469322    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4sxs\" (UniqueName: \"kubernetes.io/projected/d9a3dd23-a51f-42b8-b4d5-fb95eb681add-kube-api-access-m4sxs\") pod \"node-ca-tgdbb\" (UID: \"d9a3dd23-a51f-42b8-b4d5-fb95eb681add\") " pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.548496 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.548409    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws"
Jul 09 16:50:55.558666 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.558631    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2stnr"
Jul 09 16:50:55.570325 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.570300    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-z2hh6"
Jul 09 16:50:55.577013 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.576994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-nz2rr"
Jul 09 16:50:55.583593 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.583545    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:50:55.591184 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.591168    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:50:55.599750 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.599730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-f8c9g"
Jul 09 16:50:55.607269 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.607248    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-qkg46"
Jul 09 16:50:55.613841 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.613821    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-tgdbb"
Jul 09 16:50:55.757697 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.757668    2563 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 16:50:55.963125 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.963037    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:55.963125 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.963088    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:55.963150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963195    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963210    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963265    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963274    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs podName:dc814b9e-7680-4729-9858-c192c4de754c nodeName:}" failed. No retries permitted until 2026-07-09 16:50:56.963255373 +0000 UTC m=+4.043196335 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs") pod "network-metrics-daemon-qh9z6" (UID: "dc814b9e-7680-4729-9858-c192c4de754c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963280    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963292    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m7rsh for pod openshift-network-diagnostics/network-check-target-fvr59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963293    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret podName:ae1eeec8-a184-4829-b055-c7715ca287ff nodeName:}" failed. No retries permitted until 2026-07-09 16:50:56.963284268 +0000 UTC m=+4.043225215 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret") pod "global-pull-secret-syncer-cf425" (UID: "ae1eeec8-a184-4829-b055-c7715ca287ff") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:55.963340 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:55.963339    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh podName:fb9c8550-2a74-4e96-8a11-42c778b6d711 nodeName:}" failed. No retries permitted until 2026-07-09 16:50:56.963323809 +0000 UTC m=+4.043264760 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7rsh" (UniqueName: "kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh") pod "network-check-target-fvr59" (UID: "fb9c8550-2a74-4e96-8a11-42c778b6d711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:56.285441 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:56.285361    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:45:55 +0000 UTC" deadline="2028-02-25 01:28:32.36433965 +0000 UTC"
Jul 09 16:50:56.285441 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:56.285390    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14288h37m36.078953314s"
Jul 09 16:50:56.347786 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:56.347745    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:56.347928 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.347875    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:50:56.493308 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.493285    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39cd66d1_cdef_4ed1_b094_5d664f273801.slice/crio-8a5ced68e5501251ff5d895f6fddaa1e2b2daf6a6c393ef1bfa884a18cfd88b5 WatchSource:0}: Error finding container 8a5ced68e5501251ff5d895f6fddaa1e2b2daf6a6c393ef1bfa884a18cfd88b5: Status 404 returned error can't find the container with id 8a5ced68e5501251ff5d895f6fddaa1e2b2daf6a6c393ef1bfa884a18cfd88b5
Jul 09 16:50:56.494966 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.494844    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ae3ca7c_05a4_4e68_b815_426cc000fa72.slice/crio-563faac335f3b6b51f8df2f985eb1d882cc3e54377c6fd9cb516126a78d5a90e WatchSource:0}: Error finding container 563faac335f3b6b51f8df2f985eb1d882cc3e54377c6fd9cb516126a78d5a90e: Status 404 returned error can't find the container with id 563faac335f3b6b51f8df2f985eb1d882cc3e54377c6fd9cb516126a78d5a90e
Jul 09 16:50:56.495845 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.495823    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9e510b3_dc90_4a0e_b2fc_4e7e01602b7f.slice/crio-a4c088ca185508a62b4a730c14667f731658fb841acfe459fecefe0bae01ea04 WatchSource:0}: Error finding container a4c088ca185508a62b4a730c14667f731658fb841acfe459fecefe0bae01ea04: Status 404 returned error can't find the container with id a4c088ca185508a62b4a730c14667f731658fb841acfe459fecefe0bae01ea04
Jul 09 16:50:56.499487 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.499466    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a3dd23_a51f_42b8_b4d5_fb95eb681add.slice/crio-0a6e39d872d8fefaed59ecc93c914787dfc548bad1df25e7bfb2eccc4708cf29 WatchSource:0}: Error finding container 0a6e39d872d8fefaed59ecc93c914787dfc548bad1df25e7bfb2eccc4708cf29: Status 404 returned error can't find the container with id 0a6e39d872d8fefaed59ecc93c914787dfc548bad1df25e7bfb2eccc4708cf29
Jul 09 16:50:56.500887 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.500865    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod628ba919_14e4_4c4b_928c_f3518abff7e1.slice/crio-8efc1efb8c28608232b3caa3248ad26a641ba5c9cd3dda6e750772a2f83e9f0a WatchSource:0}: Error finding container 8efc1efb8c28608232b3caa3248ad26a641ba5c9cd3dda6e750772a2f83e9f0a: Status 404 returned error can't find the container with id 8efc1efb8c28608232b3caa3248ad26a641ba5c9cd3dda6e750772a2f83e9f0a
Jul 09 16:50:56.502303 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.502254    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ef3d43_c5db_49ef_8141_5cfbd35f397a.slice/crio-b7ed1c1bdbd9f89c6022528187c357fa242c6c9e41ed2bf55053bec0109a81e7 WatchSource:0}: Error finding container b7ed1c1bdbd9f89c6022528187c357fa242c6c9e41ed2bf55053bec0109a81e7: Status 404 returned error can't find the container with id b7ed1c1bdbd9f89c6022528187c357fa242c6c9e41ed2bf55053bec0109a81e7
Jul 09 16:50:56.503229 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.503207    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75f17ac6_afdd_4ff0_8345_d11aff9e54b6.slice/crio-e418ddfd9aa0784789f98b16e9cfbda254687e5363cc81b3b7978eb14eb687a5 WatchSource:0}: Error finding container e418ddfd9aa0784789f98b16e9cfbda254687e5363cc81b3b7978eb14eb687a5: Status 404 returned error can't find the container with id e418ddfd9aa0784789f98b16e9cfbda254687e5363cc81b3b7978eb14eb687a5
Jul 09 16:50:56.504375 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.504347    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda58a8df2_224f_4576_9d55_3fc4486f00f0.slice/crio-fe806deee4568624b47142aa40a047eab8be7a0adaa3c6a52c53452163daa690 WatchSource:0}: Error finding container fe806deee4568624b47142aa40a047eab8be7a0adaa3c6a52c53452163daa690: Status 404 returned error can't find the container with id fe806deee4568624b47142aa40a047eab8be7a0adaa3c6a52c53452163daa690
Jul 09 16:50:56.506930 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:50:56.505940    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57af9d2b_1213_446a_9d46_f43546d8603e.slice/crio-85b989cfeb05dc7b8044df2f588f41e577b17850efcd4b86a798dd78f3dd6ff3 WatchSource:0}: Error finding container 85b989cfeb05dc7b8044df2f588f41e577b17850efcd4b86a798dd78f3dd6ff3: Status 404 returned error can't find the container with id 85b989cfeb05dc7b8044df2f588f41e577b17850efcd4b86a798dd78f3dd6ff3
Jul 09 16:50:56.969924 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:56.969569    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:56.969924 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:56.969882    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:56.969924 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.969708    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:50:56.969924 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:56.969916    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:56.970194 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.969937    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:50:56.970194 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.969955    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m7rsh for pod openshift-network-diagnostics/network-check-target-fvr59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:56.970194 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.970020    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh podName:fb9c8550-2a74-4e96-8a11-42c778b6d711 nodeName:}" failed. No retries permitted until 2026-07-09 16:50:58.969999813 +0000 UTC m=+6.049940772 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7rsh" (UniqueName: "kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh") pod "network-check-target-fvr59" (UID: "fb9c8550-2a74-4e96-8a11-42c778b6d711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:56.970194 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.970048    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:56.970194 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.970098    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret podName:ae1eeec8-a184-4829-b055-c7715ca287ff nodeName:}" failed. No retries permitted until 2026-07-09 16:50:58.970083278 +0000 UTC m=+6.050024239 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret") pod "global-pull-secret-syncer-cf425" (UID: "ae1eeec8-a184-4829-b055-c7715ca287ff") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:56.970194 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.970158    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:56.970449 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:56.970207    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs podName:dc814b9e-7680-4729-9858-c192c4de754c nodeName:}" failed. No retries permitted until 2026-07-09 16:50:58.970193334 +0000 UTC m=+6.050134294 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs") pod "network-metrics-daemon-qh9z6" (UID: "dc814b9e-7680-4729-9858-c192c4de754c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:57.285804 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.285691    2563 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 16:45:55 +0000 UTC" deadline="2028-03-30 21:48:37.85029089 +0000 UTC"
Jul 09 16:50:57.285804 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.285754    2563 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15124h57m40.564541424s"
Jul 09 16:50:57.350411 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.350381    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:57.350574 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:57.350519    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:50:57.350936 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.350912    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:57.351043 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:57.351020    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:50:57.373218 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.373186    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgdbb" event={"ID":"d9a3dd23-a51f-42b8-b4d5-fb95eb681add","Type":"ContainerStarted","Data":"0a6e39d872d8fefaed59ecc93c914787dfc548bad1df25e7bfb2eccc4708cf29"}
Jul 09 16:50:57.376695 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.376665    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z2hh6" event={"ID":"4ae3ca7c-05a4-4e68-b815-426cc000fa72","Type":"ContainerStarted","Data":"563faac335f3b6b51f8df2f985eb1d882cc3e54377c6fd9cb516126a78d5a90e"}
Jul 09 16:50:57.387052 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.386999    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qkg46" event={"ID":"39cd66d1-cdef-4ed1-b094-5d664f273801","Type":"ContainerStarted","Data":"8a5ced68e5501251ff5d895f6fddaa1e2b2daf6a6c393ef1bfa884a18cfd88b5"}
Jul 09 16:50:57.399181 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.399153    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" event={"ID":"fc2d5eb401e4b3f52c292a48eec32885","Type":"ContainerStarted","Data":"b0052c0ded58e58100de48c1462b62b6be7480a41c8b8e695c9a247d3ada609d"}
Jul 09 16:50:57.412705 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.407403    2563 generic.go:356] "Generic (PLEG): container finished" podID="43807d716b03976d3a2bd26c692f140f" containerID="aa165cebdc27c858025b06156d6f4d8e54da6a2fb603d4840aca0f71b6d5edf5" exitCode=0
Jul 09 16:50:57.412705 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.407477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"43807d716b03976d3a2bd26c692f140f","Type":"ContainerDied","Data":"aa165cebdc27c858025b06156d6f4d8e54da6a2fb603d4840aca0f71b6d5edf5"}
Jul 09 16:50:57.424438 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.424411    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws" event={"ID":"57af9d2b-1213-446a-9d46-f43546d8603e","Type":"ContainerStarted","Data":"85b989cfeb05dc7b8044df2f588f41e577b17850efcd4b86a798dd78f3dd6ff3"}
Jul 09 16:50:57.431719 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.431668    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-139-29.ec2.internal" podStartSLOduration=3.431627431 podStartE2EDuration="3.431627431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:50:57.415751211 +0000 UTC m=+4.495692182" watchObservedRunningTime="2026-07-09 16:50:57.431627431 +0000 UTC m=+4.511568400"
Jul 09 16:50:57.440248 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.440099    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ccpn" event={"ID":"20ef3d43-c5db-49ef-8141-5cfbd35f397a","Type":"ContainerStarted","Data":"b7ed1c1bdbd9f89c6022528187c357fa242c6c9e41ed2bf55053bec0109a81e7"}
Jul 09 16:50:57.446880 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.446855    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nz2rr" event={"ID":"a58a8df2-224f-4576-9d55-3fc4486f00f0","Type":"ContainerStarted","Data":"fe806deee4568624b47142aa40a047eab8be7a0adaa3c6a52c53452163daa690"}
Jul 09 16:50:57.456114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.456088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"e418ddfd9aa0784789f98b16e9cfbda254687e5363cc81b3b7978eb14eb687a5"}
Jul 09 16:50:57.469645 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.469577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerStarted","Data":"8efc1efb8c28608232b3caa3248ad26a641ba5c9cd3dda6e750772a2f83e9f0a"}
Jul 09 16:50:57.491035 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:57.491004    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8c9g" event={"ID":"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f","Type":"ContainerStarted","Data":"a4c088ca185508a62b4a730c14667f731658fb841acfe459fecefe0bae01ea04"}
Jul 09 16:50:58.348944 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:58.346883    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:58.348944 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.347019    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:50:58.527104 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:58.527021    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" event={"ID":"43807d716b03976d3a2bd26c692f140f","Type":"ContainerStarted","Data":"23332b5f37c75967a353b3536cae3d43edc039497674f68a3e18cc4ebafa88e1"}
Jul 09 16:50:58.544876 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:58.544212    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-139-29.ec2.internal" podStartSLOduration=4.544189955 podStartE2EDuration="4.544189955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:50:58.543232363 +0000 UTC m=+5.623173335" watchObservedRunningTime="2026-07-09 16:50:58.544189955 +0000 UTC m=+5.624130926"
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:58.986760    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:58.986815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:58.986847    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.986903    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.986930    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.986943    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m7rsh for pod openshift-network-diagnostics/network-check-target-fvr59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.986956    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.987000    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh podName:fb9c8550-2a74-4e96-8a11-42c778b6d711 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:02.98698114 +0000 UTC m=+10.066922090 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7rsh" (UniqueName: "kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh") pod "network-check-target-fvr59" (UID: "fb9c8550-2a74-4e96-8a11-42c778b6d711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.987019    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret podName:ae1eeec8-a184-4829-b055-c7715ca287ff nodeName:}" failed. No retries permitted until 2026-07-09 16:51:02.987009553 +0000 UTC m=+10.066950500 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret") pod "global-pull-secret-syncer-cf425" (UID: "ae1eeec8-a184-4829-b055-c7715ca287ff") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.987035    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:58.987100 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:58.987074    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs podName:dc814b9e-7680-4729-9858-c192c4de754c nodeName:}" failed. No retries permitted until 2026-07-09 16:51:02.987057869 +0000 UTC m=+10.066998829 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs") pod "network-metrics-daemon-qh9z6" (UID: "dc814b9e-7680-4729-9858-c192c4de754c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:50:59.350019 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:59.348890    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:50:59.350019 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:59.349017    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:50:59.350019 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:50:59.349795    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:50:59.350019 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:50:59.349898    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:00.347129 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:00.347094    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:00.347328 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:00.347251    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:01.349478 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:01.349365    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:01.349951 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:01.349513    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:01.349951 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:01.349623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:01.349951 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:01.349738    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:02.347909 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:02.347873    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:02.348101 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:02.348057    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:03.022148 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:03.022108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:03.022171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:03.022204    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022309    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022335    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022349    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m7rsh for pod openshift-network-diagnostics/network-check-target-fvr59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022360    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022358    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022411    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh podName:fb9c8550-2a74-4e96-8a11-42c778b6d711 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:11.022392044 +0000 UTC m=+18.102333011 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7rsh" (UniqueName: "kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh") pod "network-check-target-fvr59" (UID: "fb9c8550-2a74-4e96-8a11-42c778b6d711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022434    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret podName:ae1eeec8-a184-4829-b055-c7715ca287ff nodeName:}" failed. No retries permitted until 2026-07-09 16:51:11.022423461 +0000 UTC m=+18.102364409 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret") pod "global-pull-secret-syncer-cf425" (UID: "ae1eeec8-a184-4829-b055-c7715ca287ff") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:03.022573 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.022450    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs podName:dc814b9e-7680-4729-9858-c192c4de754c nodeName:}" failed. No retries permitted until 2026-07-09 16:51:11.022440022 +0000 UTC m=+18.102380970 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs") pod "network-metrics-daemon-qh9z6" (UID: "dc814b9e-7680-4729-9858-c192c4de754c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:03.348350 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:03.348271    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:03.348535 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.348385    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:03.348535 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:03.348428    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:03.348638 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:03.348549    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:04.347201 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:04.347154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:04.347700 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:04.347302    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:05.347993 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:05.347414    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:05.347993 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:05.347451    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:05.347993 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:05.347586    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:05.347993 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:05.347753    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:06.347164 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:06.347129    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:06.347346 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:06.347271    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:07.347823 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:07.347742    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:07.348256 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:07.347755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:07.348256 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:07.347869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:07.348256 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:07.347974    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:08.347792 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:08.347752    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:08.347986 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:08.347890    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:09.347026 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:09.346994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:09.347183 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:09.346994    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:09.347183 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:09.347153    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:09.347303 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:09.347216    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:10.347733 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:10.347691    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:10.348177 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:10.347833    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:11.080570 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:11.080529    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:11.080774 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:11.080603    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:11.080774 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:11.080643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:11.080774 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080676    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:11.080774 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080755    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret podName:ae1eeec8-a184-4829-b055-c7715ca287ff nodeName:}" failed. No retries permitted until 2026-07-09 16:51:27.080735358 +0000 UTC m=+34.160676319 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret") pod "global-pull-secret-syncer-cf425" (UID: "ae1eeec8-a184-4829-b055-c7715ca287ff") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:11.080774 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080754    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:11.081016 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080783    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:11.081016 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080805    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:11.081016 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080819    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m7rsh for pod openshift-network-diagnostics/network-check-target-fvr59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:11.081016 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080794    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs podName:dc814b9e-7680-4729-9858-c192c4de754c nodeName:}" failed. No retries permitted until 2026-07-09 16:51:27.080783361 +0000 UTC m=+34.160724322 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs") pod "network-metrics-daemon-qh9z6" (UID: "dc814b9e-7680-4729-9858-c192c4de754c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:11.081016 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.080899    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh podName:fb9c8550-2a74-4e96-8a11-42c778b6d711 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:27.080872849 +0000 UTC m=+34.160813805 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7rsh" (UniqueName: "kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh") pod "network-check-target-fvr59" (UID: "fb9c8550-2a74-4e96-8a11-42c778b6d711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:11.347182 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:11.347102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:11.347363 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:11.347102    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:11.347363 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.347241    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:11.347363 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:11.347292    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:12.347923 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:12.347877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:12.348414 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:12.348011    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:13.347867 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:13.347833    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:13.348067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:13.347875    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:13.348067 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:13.347939    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:13.348460 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:13.348083    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:14.347195 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:14.347149    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:14.347372 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:14.347278    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:15.350349 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.349635    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:15.350349 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:15.349817    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:15.350349 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.350197    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:15.350349 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:15.350305    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:15.563977 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.563730    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-nz2rr" event={"ID":"a58a8df2-224f-4576-9d55-3fc4486f00f0","Type":"ContainerStarted","Data":"07ac9599792ff9805e1de74697c72a13f086a63a3df79602321342830b7cdc25"}
Jul 09 16:51:15.566249 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.566201    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"ada95aa02fc92281a36235d331e7206b80007daa974b7c0932d217f706e99c72"}
Jul 09 16:51:15.569397 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.569340    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerStarted","Data":"8c8bcb6b65aca402e4cfca96c911fa9e8ca88ddf517029ece073b6dd094f35de"}
Jul 09 16:51:15.571265 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.571232    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-tgdbb" event={"ID":"d9a3dd23-a51f-42b8-b4d5-fb95eb681add","Type":"ContainerStarted","Data":"5d23e597ce1f26fb182e0b73026d43cb5d909ad12cc98ee551623a6bec1ec3eb"}
Jul 09 16:51:15.573771 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.573607    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-z2hh6" event={"ID":"4ae3ca7c-05a4-4e68-b815-426cc000fa72","Type":"ContainerStarted","Data":"2232ad4628362d2ed562096228cff545de0c28576fd4533cfe63f01bdb29e08f"}
Jul 09 16:51:15.575304 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.575277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-qkg46" event={"ID":"39cd66d1-cdef-4ed1-b094-5d664f273801","Type":"ContainerStarted","Data":"321ba5cea22899293fa7fcb89a34ed95ed45931a53c205781dbaa58b0faefc00"}
Jul 09 16:51:15.577215 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.577191    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws" event={"ID":"57af9d2b-1213-446a-9d46-f43546d8603e","Type":"ContainerStarted","Data":"13e4ae38442c56b4352040d5aeec0af222e97b8162d29d27b60078b94ae78761"}
Jul 09 16:51:15.580704 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.579477    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6ccpn" event={"ID":"20ef3d43-c5db-49ef-8141-5cfbd35f397a","Type":"ContainerStarted","Data":"7ee331991b3b90c1ae9178e18e92fa69808df3d54dbe796e002599c574fbe5d8"}
Jul 09 16:51:15.595667 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.595556    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6ccpn" podStartSLOduration=8.173629734 podStartE2EDuration="22.595540794s" totalImagesPullingTime="14.42191106s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:15.595401508 +0000 UTC m=+22.675342476" watchObservedRunningTime="2026-07-09 16:51:15.595540794 +0000 UTC m=+22.675481762"
Jul 09 16:51:15.596047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.596021    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-nz2rr" podStartSLOduration=3.726783339 podStartE2EDuration="22.596012925s" totalImagesPullingTime="18.869229586s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:15.581743056 +0000 UTC m=+22.661684026" watchObservedRunningTime="2026-07-09 16:51:15.596012925 +0000 UTC m=+22.675953893"
Jul 09 16:51:15.612222 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.612044    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-qkg46" podStartSLOduration=3.782404768 podStartE2EDuration="22.612028772s" totalImagesPullingTime="18.829624004s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:15.611850541 +0000 UTC m=+22.691791511" watchObservedRunningTime="2026-07-09 16:51:15.612028772 +0000 UTC m=+22.691969740"
Jul 09 16:51:15.625523 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.625477    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-tgdbb" podStartSLOduration=4.036628688 podStartE2EDuration="22.625459899s" totalImagesPullingTime="18.588831211s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:15.625031694 +0000 UTC m=+22.704972665" watchObservedRunningTime="2026-07-09 16:51:15.625459899 +0000 UTC m=+22.705400867"
Jul 09 16:51:15.663807 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.663762    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-z2hh6" podStartSLOduration=3.8375942800000002 podStartE2EDuration="22.663745487s" totalImagesPullingTime="18.826151207s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:15.662591605 +0000 UTC m=+22.742532575" watchObservedRunningTime="2026-07-09 16:51:15.663745487 +0000 UTC m=+22.743686457"
Jul 09 16:51:15.966120 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.966084    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:51:15.966752 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:15.966736    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:51:16.347582 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.347558    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:16.347749 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:16.347704    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:16.590637 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.590601    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"b13ab9bf103cbcddbb001a78b74891255391553359c362194a88ddfcfbb2b135"}
Jul 09 16:51:16.591370 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.590693    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"2df23f5c836de51d07129151cb1443141ab2df4334451d869fbfa8ecd45d5d32"}
Jul 09 16:51:16.591370 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.590726    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"c1bec0a1ec867f930c509c4d7335db88595530dbd15b759ba6ce56d7edc3489b"}
Jul 09 16:51:16.591370 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.590767    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"90c1a6c887ab04dcbfb5c83e1df935b12334fdd4a643112405e855910ed942f2"}
Jul 09 16:51:16.591370 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.590794    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"15fb787b2c295100ca87e08d4c1d6867c3c782432883f395c5f257e15ea2a623"}
Jul 09 16:51:16.591954 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.591934    2563 generic.go:356] "Generic (PLEG): container finished" podID="628ba919-14e4-4c4b-928c-f3518abff7e1" containerID="8c8bcb6b65aca402e4cfca96c911fa9e8ca88ddf517029ece073b6dd094f35de" exitCode=0
Jul 09 16:51:16.592010 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.591996    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerDied","Data":"8c8bcb6b65aca402e4cfca96c911fa9e8ca88ddf517029ece073b6dd094f35de"}
Jul 09 16:51:16.593428 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.593366    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-f8c9g" event={"ID":"f9e510b3-dc90-4a0e-b2fc-4e7e01602b7f","Type":"ContainerStarted","Data":"db6a4bdf96444cc38a53df0e60b64603ffe59fd888773aa39decb8b3027c99c4"}
Jul 09 16:51:16.594236 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.594212    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:51:16.594336 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.594278    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6ccpn"
Jul 09 16:51:16.640390 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:16.640340    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-f8c9g" podStartSLOduration=4.815099128 podStartE2EDuration="23.640325216s" totalImagesPullingTime="18.825226088s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:16.640159286 +0000 UTC m=+23.720100254" watchObservedRunningTime="2026-07-09 16:51:16.640325216 +0000 UTC m=+23.720266185"
Jul 09 16:51:17.191720 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:17.191683    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:17.274268 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:17.274237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:17.347682 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:17.347646    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:17.347682 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:17.347678    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:17.347915 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:17.347748    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:17.347915 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:17.347869    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:17.472060 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:17.472033    2563 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 16:51:17.597485 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:17.597441    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws" event={"ID":"57af9d2b-1213-446a-9d46-f43546d8603e","Type":"ContainerStarted","Data":"f86a399f3c33d23338445d26d8d660eb0cb33da161fabf7d5f796fe2829db1a6"}
Jul 09 16:51:18.315217 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:18.315096    2563 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T16:51:17.472049856Z","UUID":"44a28caf-c4eb-4537-a7b2-534d0b34d856","Handler":null,"Name":"","Endpoint":""}
Jul 09 16:51:18.317087 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:18.317061    2563 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 16:51:18.317225 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:18.317098    2563 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 16:51:18.347824 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:18.347755    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:18.347978 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:18.347860    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:18.604235 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:18.604139    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"172a0b7ce082b013f76b604241264bcbfcfdb4f17a70f3a0a4b5362b2b19e3b3"}
Jul 09 16:51:19.347137 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:19.346957    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:19.347344 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:19.347035    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:19.347344 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:19.347224    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:19.347439 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:19.347337    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:19.608392 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:19.608299    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws" event={"ID":"57af9d2b-1213-446a-9d46-f43546d8603e","Type":"ContainerStarted","Data":"03208e23a7c83fc8a82447134874114f1014281795f968537b3016a349e7d5f3"}
Jul 09 16:51:19.629664 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:19.629599    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qgcws" podStartSLOduration=4.711929746 podStartE2EDuration="26.629579515s" totalImagesPullingTime="21.917649769s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:19.628375163 +0000 UTC m=+26.708316131" watchObservedRunningTime="2026-07-09 16:51:19.629579515 +0000 UTC m=+26.709520480"
Jul 09 16:51:20.346858 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:20.346835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:20.347004 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:20.346932    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:20.614580 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:20.614501    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" event={"ID":"75f17ac6-afdd-4ff0-8345-d11aff9e54b6","Type":"ContainerStarted","Data":"f77232f46ed3576c16b96cd793bb285d03a6307f0ff4ca801742144afd236fe7"}
Jul 09 16:51:20.645236 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:20.645021    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp" podStartSLOduration=8.753310678 podStartE2EDuration="27.645003413s" totalImagesPullingTime="18.891692735s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:20.643271514 +0000 UTC m=+27.723212482" watchObservedRunningTime="2026-07-09 16:51:20.645003413 +0000 UTC m=+27.724944382"
Jul 09 16:51:21.347097 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:21.346989    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:21.347247 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:21.347108    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:21.347247 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:21.347148    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:21.347313 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:21.347251    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:21.618246 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:21.618162    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:51:21.618703 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:21.618286    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:51:21.618800 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:21.618778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:51:21.641280 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:21.641240    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:51:21.641434 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:21.641414    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:51:22.173638 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.173604    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:22.299868 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.299837    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cf425"]
Jul 09 16:51:22.300019 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.299962    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:22.300088 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:22.300056    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:22.302590 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.302516    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qh9z6"]
Jul 09 16:51:22.302745 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.302639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:22.302816 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:22.302766    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:22.303545 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.303512    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fvr59"]
Jul 09 16:51:22.303682 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.303591    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:22.303763 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:22.303716    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:22.828156 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:22.828114    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:23.347803 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:23.347774    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:23.347956 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:23.347853    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:23.533605 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:23.533574    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:23.603027 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:23.602950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:23.703993 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:23.703963    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:23.805044 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:23.804840    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:23.872502 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:23.872477    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:24.347276 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:24.347251    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:24.347276 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:24.347277    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:24.347453 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:24.347377    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:24.347535 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:24.347517    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:24.622315 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:24.622219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:24.624731 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:24.624707    2563 generic.go:356] "Generic (PLEG): container finished" podID="628ba919-14e4-4c4b-928c-f3518abff7e1" containerID="196f5c8bd0edcd2d5017f043981160c4ef28c23742d0bd9627def98a6bc6fe0c" exitCode=0
Jul 09 16:51:24.624851 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:24.624742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerDied","Data":"196f5c8bd0edcd2d5017f043981160c4ef28c23742d0bd9627def98a6bc6fe0c"}
Jul 09 16:51:24.678149 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:24.678119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:24.944254 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:24.944213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:25.044056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:25.044022    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:25.347162 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:25.347088    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:25.347314 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:25.347244    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:25.629150 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:25.629063    2563 generic.go:356] "Generic (PLEG): container finished" podID="628ba919-14e4-4c4b-928c-f3518abff7e1" containerID="83955bcff929f0eba54da23e99d8de7237944c9d0b1e01cff1516215e0fd8651" exitCode=0
Jul 09 16:51:25.629150 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:25.629102    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerDied","Data":"83955bcff929f0eba54da23e99d8de7237944c9d0b1e01cff1516215e0fd8651"}
Jul 09 16:51:26.347432 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:26.347403    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:26.347976 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:26.347508    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh9z6" podUID="dc814b9e-7680-4729-9858-c192c4de754c"
Jul 09 16:51:26.347976 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:26.347559    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:26.347976 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:26.347668    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-cf425" podUID="ae1eeec8-a184-4829-b055-c7715ca287ff"
Jul 09 16:51:26.366725 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:26.366673    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:26.479425 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:26.479383    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:27.101827 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.101793    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:27.101827 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.101840    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.101863    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.101958    2563 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.101975    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.101996    2563 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.102010    2563 projected.go:196] Error preparing data for projected volume kube-api-access-m7rsh for pod openshift-network-diagnostics/network-check-target-fvr59: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.102027    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs podName:dc814b9e-7680-4729-9858-c192c4de754c nodeName:}" failed. No retries permitted until 2026-07-09 16:51:59.102009283 +0000 UTC m=+66.181950231 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs") pod "network-metrics-daemon-qh9z6" (UID: "dc814b9e-7680-4729-9858-c192c4de754c") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.101957    2563 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:27.102063 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.102060    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh podName:fb9c8550-2a74-4e96-8a11-42c778b6d711 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:59.102043859 +0000 UTC m=+66.181984823 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-m7rsh" (UniqueName: "kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh") pod "network-check-target-fvr59" (UID: "fb9c8550-2a74-4e96-8a11-42c778b6d711") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 16:51:27.102337 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.102103    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret podName:ae1eeec8-a184-4829-b055-c7715ca287ff nodeName:}" failed. No retries permitted until 2026-07-09 16:51:59.102087706 +0000 UTC m=+66.182028669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret") pod "global-pull-secret-syncer-cf425" (UID: "ae1eeec8-a184-4829-b055-c7715ca287ff") : object "kube-system"/"original-pull-secret" not registered
Jul 09 16:51:27.231679 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.231635    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:27.330704 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.330674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:27.346958 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.346936    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:27.347060 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:27.347042    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fvr59" podUID="fb9c8550-2a74-4e96-8a11-42c778b6d711"
Jul 09 16:51:27.431197 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.431150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:27.489210 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.489176    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:27.636362 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.636277    2563 generic.go:356] "Generic (PLEG): container finished" podID="628ba919-14e4-4c4b-928c-f3518abff7e1" containerID="1e1030b28c867c1b46caeae6c1d1944486a748b99b77ae6d41ad124a2c6f90d4" exitCode=0
Jul 09 16:51:27.636362 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:27.636332    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerDied","Data":"1e1030b28c867c1b46caeae6c1d1944486a748b99b77ae6d41ad124a2c6f90d4"}
Jul 09 16:51:28.307186 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.307150    2563 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-139-29.ec2.internal" event="NodeReady"
Jul 09 16:51:28.307453 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.307317    2563 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 16:51:28.347943 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.347904    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:28.348123 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.347906    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:28.351312 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.351287    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 16:51:28.351312 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.351307    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4tqb9\""
Jul 09 16:51:28.351507 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.351288    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 16:51:28.359572 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.359430    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cpsnp"]
Jul 09 16:51:28.362574 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.362553    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-64d7b4644d-ctptj"]
Jul 09 16:51:28.362706 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.362694    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.365903 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.365859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 16:51:28.366434 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.366394    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-h84wl\""
Jul 09 16:51:28.366726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.366707    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 16:51:28.367038 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.367020    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 16:51:28.368558 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.368536    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-wgwn9"]
Jul 09 16:51:28.368705 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.368686    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.371214 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.371193    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 16:51:28.371730 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.371626    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpsnp"]
Jul 09 16:51:28.371886 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.371869    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.372275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.371666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 16:51:28.372403 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.372389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 16:51:28.372786 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.372406    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-2wj2f\""
Jul 09 16:51:28.375547 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.375529    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 16:51:28.375826 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.375639    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64d7b4644d-ctptj"]
Jul 09 16:51:28.375917 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.375738    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 16:51:28.375917 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.375772    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-8vwg8\""
Jul 09 16:51:28.378322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.378305    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 16:51:28.386337 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.386231    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wgwn9"]
Jul 09 16:51:28.504614 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.504580    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:28.512402 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxkvd\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-kube-api-access-cxkvd\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512537 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512417    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh8dm\" (UniqueName: \"kubernetes.io/projected/7f9225d8-e78f-4897-aeb8-d382e7f9b718-kube-api-access-mh8dm\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.512537 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512445    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-registry-certificates\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512537 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512488    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-image-registry-private-configuration\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512537 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512526    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b12220bf-d552-4c7f-b847-213503190981-ca-trust-extracted\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512729 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512616    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-trusted-ca\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512729 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.512729 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512724    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbtsj\" (UniqueName: \"kubernetes.io/projected/055cde3f-5066-4497-8676-304897be9ff2-kube-api-access-cbtsj\") pod \"ingress-canary-cpsnp\" (UID: \"055cde3f-5066-4497-8676-304897be9ff2\") " pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.512857 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512745    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-bound-sa-token\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512857 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512797    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055cde3f-5066-4497-8676-304897be9ff2-cert\") pod \"ingress-canary-cpsnp\" (UID: \"055cde3f-5066-4497-8676-304897be9ff2\") " pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.512857 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512836    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9225d8-e78f-4897-aeb8-d382e7f9b718-tmp-dir\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.512968 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512886    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512968 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-installation-pull-secrets\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.512968 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.512932    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9225d8-e78f-4897-aeb8-d382e7f9b718-config-volume\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.574843 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.574766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:28.614220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614182    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-installation-pull-secrets\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614220 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614218    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9225d8-e78f-4897-aeb8-d382e7f9b718-config-volume\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614240    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxkvd\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-kube-api-access-cxkvd\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614262    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh8dm\" (UniqueName: \"kubernetes.io/projected/7f9225d8-e78f-4897-aeb8-d382e7f9b718-kube-api-access-mh8dm\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614289    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-registry-certificates\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614325    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-image-registry-private-configuration\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614349    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b12220bf-d552-4c7f-b847-213503190981-ca-trust-extracted\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614375    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-trusted-ca\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614398    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.614443 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbtsj\" (UniqueName: \"kubernetes.io/projected/055cde3f-5066-4497-8676-304897be9ff2-kube-api-access-cbtsj\") pod \"ingress-canary-cpsnp\" (UID: \"055cde3f-5066-4497-8676-304897be9ff2\") " pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.614865 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-bound-sa-token\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614865 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614504    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055cde3f-5066-4497-8676-304897be9ff2-cert\") pod \"ingress-canary-cpsnp\" (UID: \"055cde3f-5066-4497-8676-304897be9ff2\") " pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.614865 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614543    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9225d8-e78f-4897-aeb8-d382e7f9b718-tmp-dir\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.614865 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614636    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.614865 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:28.614788    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 16:51:28.614865 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:28.614808    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64d7b4644d-ctptj: secret "image-registry-tls" not found
Jul 09 16:51:28.615137 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:28.614878    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls podName:b12220bf-d552-4c7f-b847-213503190981 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:29.114859318 +0000 UTC m=+36.194800268 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls") pod "image-registry-64d7b4644d-ctptj" (UID: "b12220bf-d552-4c7f-b847-213503190981") : secret "image-registry-tls" not found
Jul 09 16:51:28.615137 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.614930    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7f9225d8-e78f-4897-aeb8-d382e7f9b718-config-volume\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.615137 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:28.614954    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:51:28.615137 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:28.615035    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls podName:7f9225d8-e78f-4897-aeb8-d382e7f9b718 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:29.11501422 +0000 UTC m=+36.194955182 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls") pod "dns-default-wgwn9" (UID: "7f9225d8-e78f-4897-aeb8-d382e7f9b718") : secret "dns-default-metrics-tls" not found
Jul 09 16:51:28.615137 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.615123    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-registry-certificates\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.615375 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.615215    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7f9225d8-e78f-4897-aeb8-d382e7f9b718-tmp-dir\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.615428 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.615404    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b12220bf-d552-4c7f-b847-213503190981-ca-trust-extracted\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.616027 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.616002    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-trusted-ca\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.618747 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.618727    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-installation-pull-secrets\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.618747 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.618739    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/055cde3f-5066-4497-8676-304897be9ff2-cert\") pod \"ingress-canary-cpsnp\" (UID: \"055cde3f-5066-4497-8676-304897be9ff2\") " pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.618861 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.618767    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-image-registry-private-configuration\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.629361 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.629335    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-bound-sa-token\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.629705 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.629688    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbtsj\" (UniqueName: \"kubernetes.io/projected/055cde3f-5066-4497-8676-304897be9ff2-kube-api-access-cbtsj\") pod \"ingress-canary-cpsnp\" (UID: \"055cde3f-5066-4497-8676-304897be9ff2\") " pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.629763 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.629736    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxkvd\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-kube-api-access-cxkvd\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:28.630644 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.630626    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh8dm\" (UniqueName: \"kubernetes.io/projected/7f9225d8-e78f-4897-aeb8-d382e7f9b718-kube-api-access-mh8dm\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:28.679551 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.679519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cpsnp"
Jul 09 16:51:28.823438 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:28.823201    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cpsnp"]
Jul 09 16:51:28.829351 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:28.829325    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod055cde3f_5066_4497_8676_304897be9ff2.slice/crio-ad164ecbf20e60a12871829f2ab449db4c97e04def9e9ccef25ef28a8c229a7c WatchSource:0}: Error finding container ad164ecbf20e60a12871829f2ab449db4c97e04def9e9ccef25ef28a8c229a7c: Status 404 returned error can't find the container with id ad164ecbf20e60a12871829f2ab449db4c97e04def9e9ccef25ef28a8c229a7c
Jul 09 16:51:29.118598 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:29.118502    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:29.118598 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:29.118562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:29.118835 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:29.118677    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 16:51:29.118835 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:29.118698    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64d7b4644d-ctptj: secret "image-registry-tls" not found
Jul 09 16:51:29.118835 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:29.118715    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:51:29.118835 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:29.118755    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls podName:b12220bf-d552-4c7f-b847-213503190981 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:30.118736099 +0000 UTC m=+37.198677046 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls") pod "image-registry-64d7b4644d-ctptj" (UID: "b12220bf-d552-4c7f-b847-213503190981") : secret "image-registry-tls" not found
Jul 09 16:51:29.118835 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:29.118767    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls podName:7f9225d8-e78f-4897-aeb8-d382e7f9b718 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:30.11876145 +0000 UTC m=+37.198702397 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls") pod "dns-default-wgwn9" (UID: "7f9225d8-e78f-4897-aeb8-d382e7f9b718") : secret "dns-default-metrics-tls" not found
Jul 09 16:51:29.347928 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:29.347889    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:29.351048 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:29.351002    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 16:51:29.352153 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:29.352130    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2fgzg\""
Jul 09 16:51:29.352259 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:29.352130    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 16:51:29.642142 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:29.642106    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpsnp" event={"ID":"055cde3f-5066-4497-8676-304897be9ff2","Type":"ContainerStarted","Data":"ad164ecbf20e60a12871829f2ab449db4c97e04def9e9ccef25ef28a8c229a7c"}
Jul 09 16:51:30.126470 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:30.126363    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:30.126470 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:30.126423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:30.126765 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:30.126528    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 16:51:30.126765 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:30.126536    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:51:30.126765 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:30.126625    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls podName:7f9225d8-e78f-4897-aeb8-d382e7f9b718 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:32.126600135 +0000 UTC m=+39.206541104 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls") pod "dns-default-wgwn9" (UID: "7f9225d8-e78f-4897-aeb8-d382e7f9b718") : secret "dns-default-metrics-tls" not found
Jul 09 16:51:30.126765 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:30.126542    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64d7b4644d-ctptj: secret "image-registry-tls" not found
Jul 09 16:51:30.126765 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:30.126684    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls podName:b12220bf-d552-4c7f-b847-213503190981 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:32.126673094 +0000 UTC m=+39.206614056 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls") pod "image-registry-64d7b4644d-ctptj" (UID: "b12220bf-d552-4c7f-b847-213503190981") : secret "image-registry-tls" not found
Jul 09 16:51:30.335117 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:30.335081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:30.450257 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:30.450175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:30.646116 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:30.646075    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:30.708543 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:30.708381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:31.261920 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.261885    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-m9mbt"]
Jul 09 16:51:31.264616 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.264586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:31.295210 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.295169    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m9mbt"]
Jul 09 16:51:31.295364 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.295310    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.297989 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.297968    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 16:51:31.298135 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.298043    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-jjsxm\""
Jul 09 16:51:31.298135 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.298118    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 16:51:31.298249 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.298136    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 16:51:31.298249 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.298135    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 16:51:31.324678 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.324634    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:31.436396 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.436352    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.436579 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.436406    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.436579 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.436531    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-data-volume\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.436729 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.436581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zh7kx\" (UniqueName: \"kubernetes.io/projected/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-api-access-zh7kx\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.436729 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.436624    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-crio-socket\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.537374 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.537298    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-data-volume\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.537374 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.537342    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zh7kx\" (UniqueName: \"kubernetes.io/projected/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-api-access-zh7kx\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.537374 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.537365    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-crio-socket\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.537593 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.537408    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.537593 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.537428    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.537593 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:31.537528    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 16:51:31.537593 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:31.537544    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:31.537798 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:31.537600    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm podName:a1b9d36e-47f6-43cf-af37-c3ce7f486e34 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:32.037585028 +0000 UTC m=+39.117525995 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m9mbt" (UID: "a1b9d36e-47f6-43cf-af37-c3ce7f486e34") : configmap "kube-rbac-proxy" not found
Jul 09 16:51:31.537798 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:31.537613    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls podName:a1b9d36e-47f6-43cf-af37-c3ce7f486e34 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:32.037608031 +0000 UTC m=+39.117548978 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9mbt" (UID: "a1b9d36e-47f6-43cf-af37-c3ce7f486e34") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:31.537798 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.537630    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-crio-socket\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.537977 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.537907    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-data-volume\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.552671 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.552638    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zh7kx\" (UniqueName: \"kubernetes.io/projected/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-api-access-zh7kx\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:31.925944 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.925906    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:31.950181 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.950149    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"]
Jul 09 16:51:31.953678 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.953639    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"
Jul 09 16:51:31.956842 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.956777    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 16:51:31.956961 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.956928    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 16:51:31.957880 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.957859    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-q56h7\""
Jul 09 16:51:31.967136 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:31.967113    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"]
Jul 09 16:51:32.042334 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.042284    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpn2b\" (UniqueName: \"kubernetes.io/projected/17eafe93-1fc4-426a-85f8-4ee1c9e2136b-kube-api-access-gpn2b\") pod \"migrator-7f59b54f6c-kj7cd\" (UID: \"17eafe93-1fc4-426a-85f8-4ee1c9e2136b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"
Jul 09 16:51:32.042542 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.042405    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:32.042542 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.042452    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:32.042542 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.042536    2563 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 16:51:32.042830 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.042596    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm podName:a1b9d36e-47f6-43cf-af37-c3ce7f486e34 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:33.042576656 +0000 UTC m=+40.122517607 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm") pod "insights-runtime-extractor-m9mbt" (UID: "a1b9d36e-47f6-43cf-af37-c3ce7f486e34") : configmap "kube-rbac-proxy" not found
Jul 09 16:51:32.043160 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.043062    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:32.043160 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.043140    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls podName:a1b9d36e-47f6-43cf-af37-c3ce7f486e34 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:33.043106637 +0000 UTC m=+40.123047602 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9mbt" (UID: "a1b9d36e-47f6-43cf-af37-c3ce7f486e34") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:32.143751 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.143663    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:32.143918 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.143758    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:32.143918 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.143803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpn2b\" (UniqueName: \"kubernetes.io/projected/17eafe93-1fc4-426a-85f8-4ee1c9e2136b-kube-api-access-gpn2b\") pod \"migrator-7f59b54f6c-kj7cd\" (UID: \"17eafe93-1fc4-426a-85f8-4ee1c9e2136b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"
Jul 09 16:51:32.143918 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.143828    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 16:51:32.143918 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.143852    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64d7b4644d-ctptj: secret "image-registry-tls" not found
Jul 09 16:51:32.143918 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.143918    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls podName:b12220bf-d552-4c7f-b847-213503190981 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:36.143897721 +0000 UTC m=+43.223838685 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls") pod "image-registry-64d7b4644d-ctptj" (UID: "b12220bf-d552-4c7f-b847-213503190981") : secret "image-registry-tls" not found
Jul 09 16:51:32.144161 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.143924    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:51:32.144161 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:32.143981    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls podName:7f9225d8-e78f-4897-aeb8-d382e7f9b718 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:36.143965008 +0000 UTC m=+43.223905956 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls") pod "dns-default-wgwn9" (UID: "7f9225d8-e78f-4897-aeb8-d382e7f9b718") : secret "dns-default-metrics-tls" not found
Jul 09 16:51:32.153099 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.153076    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpn2b\" (UniqueName: \"kubernetes.io/projected/17eafe93-1fc4-426a-85f8-4ee1c9e2136b-kube-api-access-gpn2b\") pod \"migrator-7f59b54f6c-kj7cd\" (UID: \"17eafe93-1fc4-426a-85f8-4ee1c9e2136b\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"
Jul 09 16:51:32.264145 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.264110    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"
Jul 09 16:51:32.873172 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:32.873136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:33.002713 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.002585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:33.051469 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.051436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:33.051469 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.051482    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:33.051886 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:33.051860    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:33.052000 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:33.051945    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls podName:a1b9d36e-47f6-43cf-af37-c3ce7f486e34 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:35.05192454 +0000 UTC m=+42.131865503 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9mbt" (UID: "a1b9d36e-47f6-43cf-af37-c3ce7f486e34") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:33.052106 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.052089    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:33.057825 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.057774    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qkg46_39cd66d1-cdef-4ed1-b094-5d664f273801/dns-node-resolver/0.log"
Jul 09 16:51:33.102565 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.102533    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:33.352340 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.352302    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd"]
Jul 09 16:51:33.369282 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.369251    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:33.381043 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:33.380999    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eafe93_1fc4_426a_85f8_4ee1c9e2136b.slice/crio-8c132a6607f87d1baea04774d5a6d178b1b04b211faaf65ceca28e41b8337f08 WatchSource:0}: Error finding container 8c132a6607f87d1baea04774d5a6d178b1b04b211faaf65ceca28e41b8337f08: Status 404 returned error can't find the container with id 8c132a6607f87d1baea04774d5a6d178b1b04b211faaf65ceca28e41b8337f08
Jul 09 16:51:33.429161 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.429123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:33.652423 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.652397    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cpsnp" event={"ID":"055cde3f-5066-4497-8676-304897be9ff2","Type":"ContainerStarted","Data":"baa83fca0f25c6005341a3faaf1d90eb99ed38e255a2883ee4af7d8e833f94f1"}
Jul 09 16:51:33.653603 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.653581    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd" event={"ID":"17eafe93-1fc4-426a-85f8-4ee1c9e2136b","Type":"ContainerStarted","Data":"8c132a6607f87d1baea04774d5a6d178b1b04b211faaf65ceca28e41b8337f08"}
Jul 09 16:51:33.656350 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.656328    2563 generic.go:356] "Generic (PLEG): container finished" podID="628ba919-14e4-4c4b-928c-f3518abff7e1" containerID="ffde6742f506ca387dfb994a394a37c4c553a146f4e4be90b17f9d4d5a4f35c2" exitCode=0
Jul 09 16:51:33.656430 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.656365    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerDied","Data":"ffde6742f506ca387dfb994a394a37c4c553a146f4e4be90b17f9d4d5a4f35c2"}
Jul 09 16:51:33.669870 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:33.669835    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cpsnp" podStartSLOduration=2.840837102 podStartE2EDuration="5.66982089s" totalImagesPullingTime="2.828983788s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:33.668494227 +0000 UTC m=+40.748435197" watchObservedRunningTime="2026-07-09 16:51:33.66982089 +0000 UTC m=+40.749761859"
Jul 09 16:51:34.056973 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.056907    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tgdbb_d9a3dd23-a51f-42b8-b4d5-fb95eb681add/node-ca/0.log"
Jul 09 16:51:34.456505 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.456428    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cpsnp_055cde3f-5066-4497-8676-304897be9ff2/serve-healthcheck-canary/0.log"
Jul 09 16:51:34.525911 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.525880    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-chl5c"]
Jul 09 16:51:34.547266 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.547233    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-chl5c"]
Jul 09 16:51:34.547395 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.547323    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.550088 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.550063    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 16:51:34.550218 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.550093    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 16:51:34.550284 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.550271    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 16:51:34.550402 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.550389    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-zrwxl\""
Jul 09 16:51:34.550458 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.550436    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 16:51:34.566909 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.566879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:34.620034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.620001    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:34.661757 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.661720    2563 generic.go:356] "Generic (PLEG): container finished" podID="628ba919-14e4-4c4b-928c-f3518abff7e1" containerID="c6a61bfdf326efbfebbd1f85281037af849f030419abdc798bfba8da6826b4d4" exitCode=0
Jul 09 16:51:34.661977 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.661774    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerDied","Data":"c6a61bfdf326efbfebbd1f85281037af849f030419abdc798bfba8da6826b4d4"}
Jul 09 16:51:34.664103 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.663972    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cf341310-e6a3-488b-8dfb-d021379f0d62-signing-key\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.664103 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.664045    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf341310-e6a3-488b-8dfb-d021379f0d62-signing-cabundle\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.664103 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.664099    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-66npk\" (UniqueName: \"kubernetes.io/projected/cf341310-e6a3-488b-8dfb-d021379f0d62-kube-api-access-66npk\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.765130 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.765093    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-66npk\" (UniqueName: \"kubernetes.io/projected/cf341310-e6a3-488b-8dfb-d021379f0d62-kube-api-access-66npk\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.765280 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.765226    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cf341310-e6a3-488b-8dfb-d021379f0d62-signing-key\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.765280 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.765276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf341310-e6a3-488b-8dfb-d021379f0d62-signing-cabundle\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.765907 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.765883    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/cf341310-e6a3-488b-8dfb-d021379f0d62-signing-cabundle\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.768277 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.768252    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/cf341310-e6a3-488b-8dfb-d021379f0d62-signing-key\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.773524 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.773503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-66npk\" (UniqueName: \"kubernetes.io/projected/cf341310-e6a3-488b-8dfb-d021379f0d62-kube-api-access-66npk\") pod \"service-ca-79f68b9f4f-chl5c\" (UID: \"cf341310-e6a3-488b-8dfb-d021379f0d62\") " pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:34.856243 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:34.856207    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c"
Jul 09 16:51:35.067391 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.067340    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:35.067858 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:35.067500    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:35.067858 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:35.067563    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls podName:a1b9d36e-47f6-43cf-af37-c3ce7f486e34 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:39.067545261 +0000 UTC m=+46.147486225 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9mbt" (UID: "a1b9d36e-47f6-43cf-af37-c3ce7f486e34") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:35.169884 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.169855    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-chl5c"]
Jul 09 16:51:35.172761 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:35.172737    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf341310_e6a3_488b_8dfb_d021379f0d62.slice/crio-058d02fd8b2bc5534f2cb7fd7eeb3fb96531e18e80d86f8a3c36b3a0da206de8 WatchSource:0}: Error finding container 058d02fd8b2bc5534f2cb7fd7eeb3fb96531e18e80d86f8a3c36b3a0da206de8: Status 404 returned error can't find the container with id 058d02fd8b2bc5534f2cb7fd7eeb3fb96531e18e80d86f8a3c36b3a0da206de8
Jul 09 16:51:35.666366 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.666279    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd" event={"ID":"17eafe93-1fc4-426a-85f8-4ee1c9e2136b","Type":"ContainerStarted","Data":"3708a58420fe027acf636f46d138042cee9e9763f6087132a57a99cd1871e165"}
Jul 09 16:51:35.666366 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.666343    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd" event={"ID":"17eafe93-1fc4-426a-85f8-4ee1c9e2136b","Type":"ContainerStarted","Data":"92eb2bc2fb309d7f1578b12cc6ef1d8c525a35ca5a0cc4c14aec597c73496025"}
Jul 09 16:51:35.669583 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.669556    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2stnr" event={"ID":"628ba919-14e4-4c4b-928c-f3518abff7e1","Type":"ContainerStarted","Data":"7d2bd6acf7a84a5e8f58b759b3fd80ba444afba1ec9435a10b8ad95520b9e960"}
Jul 09 16:51:35.672817 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.672789    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c" event={"ID":"cf341310-e6a3-488b-8dfb-d021379f0d62","Type":"ContainerStarted","Data":"058d02fd8b2bc5534f2cb7fd7eeb3fb96531e18e80d86f8a3c36b3a0da206de8"}
Jul 09 16:51:35.686771 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.686721    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-kj7cd" podStartSLOduration=3.03628635 podStartE2EDuration="4.686705672s" totalImagesPullingTime="1.650419322s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:35.685813972 +0000 UTC m=+42.765754941" watchObservedRunningTime="2026-07-09 16:51:35.686705672 +0000 UTC m=+42.766646619"
Jul 09 16:51:35.795151 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:35.795109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:36.076439 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:36.076309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:36.176854 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:36.176815    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:36.177047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:36.176885    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:36.177047 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:36.176982    2563 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 16:51:36.177047 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:36.176994    2563 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 16:51:36.177193 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:36.177066    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls podName:7f9225d8-e78f-4897-aeb8-d382e7f9b718 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:44.177044652 +0000 UTC m=+51.256985610 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls") pod "dns-default-wgwn9" (UID: "7f9225d8-e78f-4897-aeb8-d382e7f9b718") : secret "dns-default-metrics-tls" not found
Jul 09 16:51:36.177193 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:36.177001    2563 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-64d7b4644d-ctptj: secret "image-registry-tls" not found
Jul 09 16:51:36.177193 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:36.177127    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls podName:b12220bf-d552-4c7f-b847-213503190981 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:44.177113673 +0000 UTC m=+51.257054623 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls") pod "image-registry-64d7b4644d-ctptj" (UID: "b12220bf-d552-4c7f-b847-213503190981") : secret "image-registry-tls" not found
Jul 09 16:51:39.102087 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:39.102040    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:39.102463 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:39.102190    2563 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:39.102463 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:51:39.102256    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls podName:a1b9d36e-47f6-43cf-af37-c3ce7f486e34 nodeName:}" failed. No retries permitted until 2026-07-09 16:51:47.102239788 +0000 UTC m=+54.182180735 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls") pod "insights-runtime-extractor-m9mbt" (UID: "a1b9d36e-47f6-43cf-af37-c3ce7f486e34") : secret "insights-runtime-extractor-tls" not found
Jul 09 16:51:39.418792 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:39.418760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:39.683262 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:39.683182    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c" event={"ID":"cf341310-e6a3-488b-8dfb-d021379f0d62","Type":"ContainerStarted","Data":"02b7f17b553c57a01701eef5fac334b2e76e1cb10f6bc6a4125528583b0f349f"}
Jul 09 16:51:39.700171 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:39.700124    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-chl5c" podStartSLOduration=2.294831824 podStartE2EDuration="5.700109397s" totalImagesPullingTime="3.405277573s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:39.699620196 +0000 UTC m=+46.779561180" watchObservedRunningTime="2026-07-09 16:51:39.700109397 +0000 UTC m=+46.780050343"
Jul 09 16:51:39.700296 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:39.700277    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2stnr" podStartSLOduration=12.86138097 podStartE2EDuration="46.700272473s" totalImagesPullingTime="33.838891503s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:35.717422095 +0000 UTC m=+42.797363063" watchObservedRunningTime="2026-07-09 16:51:39.700272473 +0000 UTC m=+46.780213497"
Jul 09 16:51:39.868441 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:39.868408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:40.217245 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:40.217215    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:40.318355 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:40.318310    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:40.425256 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:40.425223    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:40.593213 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:40.593112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:41.600467 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:41.600431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:41.963840 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:41.963794    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:42.186361 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:42.186325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:42.367282 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:42.367245    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:42.771194 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:42.771092    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:43.562531 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:43.562492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:43.662407 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:43.662377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:43.921295 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:43.921263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:44.020868 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.020833    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:44.166075 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.166040    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:44.245602 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.245524    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:44.245602 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.245581    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:44.247885 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.247857    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7f9225d8-e78f-4897-aeb8-d382e7f9b718-metrics-tls\") pod \"dns-default-wgwn9\" (UID: \"7f9225d8-e78f-4897-aeb8-d382e7f9b718\") " pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:44.248011 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.247960    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"image-registry-64d7b4644d-ctptj\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") " pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:44.287245 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.287213    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:44.293377 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.293347    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:44.365019 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.364928    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:44.431467 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.431415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-64d7b4644d-ctptj"]
Jul 09 16:51:44.434708 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.434681    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:44.434822 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:44.434770    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb12220bf_d552_4c7f_b847_213503190981.slice/crio-8be5660226a6b64c872f74bdd0889ace7893aa2fad4b7218a7a304a9a1219a13 WatchSource:0}: Error finding container 8be5660226a6b64c872f74bdd0889ace7893aa2fad4b7218a7a304a9a1219a13: Status 404 returned error can't find the container with id 8be5660226a6b64c872f74bdd0889ace7893aa2fad4b7218a7a304a9a1219a13
Jul 09 16:51:44.446885 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.446860    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-wgwn9"]
Jul 09 16:51:44.450138 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:44.450110    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f9225d8_e78f_4897_aeb8_d382e7f9b718.slice/crio-8513d9d01fd4ef9f1d6bbb0260cca4efc278f862480f1de7576522a9398d3d65 WatchSource:0}: Error finding container 8513d9d01fd4ef9f1d6bbb0260cca4efc278f862480f1de7576522a9398d3d65: Status 404 returned error can't find the container with id 8513d9d01fd4ef9f1d6bbb0260cca4efc278f862480f1de7576522a9398d3d65
Jul 09 16:51:44.697345 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.697300    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wgwn9" event={"ID":"7f9225d8-e78f-4897-aeb8-d382e7f9b718","Type":"ContainerStarted","Data":"8513d9d01fd4ef9f1d6bbb0260cca4efc278f862480f1de7576522a9398d3d65"}
Jul 09 16:51:44.698768 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.698742    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj" event={"ID":"b12220bf-d552-4c7f-b847-213503190981","Type":"ContainerStarted","Data":"1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728"}
Jul 09 16:51:44.698914 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.698780    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj" event={"ID":"b12220bf-d552-4c7f-b847-213503190981","Type":"ContainerStarted","Data":"8be5660226a6b64c872f74bdd0889ace7893aa2fad4b7218a7a304a9a1219a13"}
Jul 09 16:51:44.698914 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.698871    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:51:44.717981 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.717924    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj" podStartSLOduration=21.717905897 podStartE2EDuration="21.717905897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:44.717109414 +0000 UTC m=+51.797050420" watchObservedRunningTime="2026-07-09 16:51:44.717905897 +0000 UTC m=+51.797846865"
Jul 09 16:51:44.770416 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:44.770309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:45.323624 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:45.323586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:45.425163 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:45.425104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:45.765293 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:45.765253    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:46.170410 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:46.170377    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:46.228699 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:46.228647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:46.570718 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:46.570624    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:46.765769 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:46.765489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:47.170558 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:47.170447    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:47.171151 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:47.171116    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:47.173383 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:47.173360    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a1b9d36e-47f6-43cf-af37-c3ce7f486e34-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-m9mbt\" (UID: \"a1b9d36e-47f6-43cf-af37-c3ce7f486e34\") " pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:47.206161 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:47.206128    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-m9mbt"
Jul 09 16:51:47.566121 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:47.566033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:47.746131 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:47.746103    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-m9mbt"]
Jul 09 16:51:47.749423 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:47.749397    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b9d36e_47f6_43cf_af37_c3ce7f486e34.slice/crio-f0b2e1c31f7b7a5c71c96c15c238e32d7393cc81ab68fd468ad41a176b079345 WatchSource:0}: Error finding container f0b2e1c31f7b7a5c71c96c15c238e32d7393cc81ab68fd468ad41a176b079345: Status 404 returned error can't find the container with id f0b2e1c31f7b7a5c71c96c15c238e32d7393cc81ab68fd468ad41a176b079345
Jul 09 16:51:47.851232 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:47.851186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:48.380875 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.380839    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:48.575118 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.575065    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:48.714308 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.714216    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wgwn9" event={"ID":"7f9225d8-e78f-4897-aeb8-d382e7f9b718","Type":"ContainerStarted","Data":"80c4d996c2e7e7809321a85bd8a88a418fd799d0308e0bd2f7b414ecc4fb9ec7"}
Jul 09 16:51:48.714308 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.714273    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-wgwn9" event={"ID":"7f9225d8-e78f-4897-aeb8-d382e7f9b718","Type":"ContainerStarted","Data":"6bdc6edfc7d67d44a7b14a5fe535b5995998136f5f45460e8ae7118525301670"}
Jul 09 16:51:48.714529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.714324    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:48.715971 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.715940    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9mbt" event={"ID":"a1b9d36e-47f6-43cf-af37-c3ce7f486e34","Type":"ContainerStarted","Data":"f0b2e1c31f7b7a5c71c96c15c238e32d7393cc81ab68fd468ad41a176b079345"}
Jul 09 16:51:48.732545 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.732439    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-wgwn9" podStartSLOduration=17.446743064 podStartE2EDuration="20.732421097s" totalImagesPullingTime="3.285678033s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:48.732186181 +0000 UTC m=+55.812127152" watchObservedRunningTime="2026-07-09 16:51:48.732421097 +0000 UTC m=+55.812362068"
Jul 09 16:51:48.970771 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:48.969558    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:49.172387 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:49.172304    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:49.567970 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:49.567935    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:49.772418 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:49.772382    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:50.375196 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:50.375155    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:50.564123 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:50.564080    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:51.732263 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:51.732229    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9mbt" event={"ID":"a1b9d36e-47f6-43cf-af37-c3ce7f486e34","Type":"ContainerStarted","Data":"fc6085a88e0a63387ba84029f1e634ef39aa19e3f7c3374e22e02ee3afa4d0f9"}
Jul 09 16:51:53.637345 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:53.637302    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5pxqp"
Jul 09 16:51:53.741051 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:53.741014    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9mbt" event={"ID":"a1b9d36e-47f6-43cf-af37-c3ce7f486e34","Type":"ContainerStarted","Data":"0e6d3b35664c8d732718fe5b94c24cc250de164b5c1c45a6567c0b7d11ef9864"}
Jul 09 16:51:54.199053 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.199012    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"]
Jul 09 16:51:54.213742 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.213707    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"]
Jul 09 16:51:54.213929 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.213840    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.218817 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.218786    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 16:51:54.218982 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.218819    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 16:51:54.220246 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.219920    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-5w4h5\""
Jul 09 16:51:54.220246 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.220020    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 16:51:54.220246 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.220149    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 16:51:54.223495 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.223469    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b0a740f-3ed3-4166-b8ec-aec444cb3ddb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6484d57b46-x6txj\" (UID: \"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.223590 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.223522    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ddhrd\" (UniqueName: \"kubernetes.io/projected/6b0a740f-3ed3-4166-b8ec-aec444cb3ddb-kube-api-access-ddhrd\") pod \"managed-serviceaccount-addon-agent-6484d57b46-x6txj\" (UID: \"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.228435 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.228406    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"]
Jul 09 16:51:54.228559 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.228452    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.228559 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.228457    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"]
Jul 09 16:51:54.231413 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.231393    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 16:51:54.231728 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.231708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 16:51:54.231837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.231708    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 16:51:54.231837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.231710    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 16:51:54.241286 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.241255    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64d7b4644d-ctptj"]
Jul 09 16:51:54.312011 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.311983    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7f5469cd45-w4zzd"]
Jul 09 16:51:54.324236 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324209    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-hub\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.324403 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324243    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q72cd\" (UniqueName: \"kubernetes.io/projected/d38a6b5d-4534-43d5-ac9a-4a951775eed5-kube-api-access-q72cd\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.324403 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324264    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-ca\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.324403 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324325    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.324403 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324367    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b0a740f-3ed3-4166-b8ec-aec444cb3ddb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6484d57b46-x6txj\" (UID: \"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.324403 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.324689 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324424    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ddhrd\" (UniqueName: \"kubernetes.io/projected/6b0a740f-3ed3-4166-b8ec-aec444cb3ddb-kube-api-access-ddhrd\") pod \"managed-serviceaccount-addon-agent-6484d57b46-x6txj\" (UID: \"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.324689 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.324549    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38a6b5d-4534-43d5-ac9a-4a951775eed5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.326830 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.326812    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/6b0a740f-3ed3-4166-b8ec-aec444cb3ddb-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-6484d57b46-x6txj\" (UID: \"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.334596 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.334574    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.338355 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.338328    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5469cd45-w4zzd"]
Jul 09 16:51:54.348749 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.348721    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ddhrd\" (UniqueName: \"kubernetes.io/projected/6b0a740f-3ed3-4166-b8ec-aec444cb3ddb-kube-api-access-ddhrd\") pod \"managed-serviceaccount-addon-agent-6484d57b46-x6txj\" (UID: \"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.357193 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.357157    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:54.425096 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38a6b5d-4534-43d5-ac9a-4a951775eed5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.425301 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425114    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49287bb3-300f-43f0-ac93-00fdf9a2fa99-registry-certificates\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425301 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425165    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-bound-sa-token\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425301 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425238    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49287bb3-300f-43f0-ac93-00fdf9a2fa99-trusted-ca\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425301 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425275    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49287bb3-300f-43f0-ac93-00fdf9a2fa99-image-registry-private-configuration\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425306    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-hub\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.425529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425330    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-88rbq\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-kube-api-access-88rbq\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425366    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q72cd\" (UniqueName: \"kubernetes.io/projected/d38a6b5d-4534-43d5-ac9a-4a951775eed5-kube-api-access-q72cd\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.425529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-ca\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.425529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425420    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49287bb3-300f-43f0-ac93-00fdf9a2fa99-ca-trust-extracted\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425461    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-registry-tls\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425528    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.425890 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425570    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49287bb3-300f-43f0-ac93-00fdf9a2fa99-installation-pull-secrets\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.425890 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425674    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.425990 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.425904    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/d38a6b5d-4534-43d5-ac9a-4a951775eed5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.428558 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.428533    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-ca\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.428726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.428629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-hub\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.428726 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.428629    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.428837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.428814    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d38a6b5d-4534-43d5-ac9a-4a951775eed5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.434300 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.434271    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q72cd\" (UniqueName: \"kubernetes.io/projected/d38a6b5d-4534-43d5-ac9a-4a951775eed5-kube-api-access-q72cd\") pod \"cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8\" (UID: \"d38a6b5d-4534-43d5-ac9a-4a951775eed5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.454305 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.454224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:54.526854 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.526809    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-registry-tls\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.526872    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49287bb3-300f-43f0-ac93-00fdf9a2fa99-installation-pull-secrets\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.526905    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49287bb3-300f-43f0-ac93-00fdf9a2fa99-registry-certificates\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.526933    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-bound-sa-token\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.526976    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49287bb3-300f-43f0-ac93-00fdf9a2fa99-trusted-ca\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.527007    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49287bb3-300f-43f0-ac93-00fdf9a2fa99-image-registry-private-configuration\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.527050    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-88rbq\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-kube-api-access-88rbq\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527230 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.527089    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49287bb3-300f-43f0-ac93-00fdf9a2fa99-ca-trust-extracted\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.527878 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.527852    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/49287bb3-300f-43f0-ac93-00fdf9a2fa99-ca-trust-extracted\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.528174 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.528148    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/49287bb3-300f-43f0-ac93-00fdf9a2fa99-registry-certificates\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.528342 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.528314    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/49287bb3-300f-43f0-ac93-00fdf9a2fa99-trusted-ca\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.529900 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.529869    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-registry-tls\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.530255 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.530232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/49287bb3-300f-43f0-ac93-00fdf9a2fa99-image-registry-private-configuration\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.530603 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.530587    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/49287bb3-300f-43f0-ac93-00fdf9a2fa99-installation-pull-secrets\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.530718 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.530698    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"
Jul 09 16:51:54.536367 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.536343    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-88rbq\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-kube-api-access-88rbq\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.536525 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.536389    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/49287bb3-300f-43f0-ac93-00fdf9a2fa99-bound-sa-token\") pod \"image-registry-7f5469cd45-w4zzd\" (UID: \"49287bb3-300f-43f0-ac93-00fdf9a2fa99\") " pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:54.539967 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.539945    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"
Jul 09 16:51:54.660359 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:54.660307    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:55.572219 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.572145    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7f5469cd45-w4zzd"]
Jul 09 16:51:55.577338 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:55.577307    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49287bb3_300f_43f0_ac93_00fdf9a2fa99.slice/crio-5aef71e640d3e62f38fcba313adef41f4f29b7f912e5ee8e93c6775277e5e079 WatchSource:0}: Error finding container 5aef71e640d3e62f38fcba313adef41f4f29b7f912e5ee8e93c6775277e5e079: Status 404 returned error can't find the container with id 5aef71e640d3e62f38fcba313adef41f4f29b7f912e5ee8e93c6775277e5e079
Jul 09 16:51:55.749582 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.749493    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd" event={"ID":"49287bb3-300f-43f0-ac93-00fdf9a2fa99","Type":"ContainerStarted","Data":"cfd57dfde3ae32aa4ff18acdd17fbc8d5c1587e7ef003c7f942404f4dbf7cef0"}
Jul 09 16:51:55.749582 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.749548    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd" event={"ID":"49287bb3-300f-43f0-ac93-00fdf9a2fa99","Type":"ContainerStarted","Data":"5aef71e640d3e62f38fcba313adef41f4f29b7f912e5ee8e93c6775277e5e079"}
Jul 09 16:51:55.750056 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.749718    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:51:55.752497 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.752455    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-m9mbt" event={"ID":"a1b9d36e-47f6-43cf-af37-c3ce7f486e34","Type":"ContainerStarted","Data":"27d1ee3faeeaabe07bfa14492f0846adebbf7d57c6455d7e42681ec5e4d79d8d"}
Jul 09 16:51:55.768783 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.768725    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd" podStartSLOduration=1.7687034320000001 podStartE2EDuration="1.768703432s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:55.76715062 +0000 UTC m=+62.847091590" watchObservedRunningTime="2026-07-09 16:51:55.768703432 +0000 UTC m=+62.848644403"
Jul 09 16:51:55.778546 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.778517    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:55.783815 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.783764    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-m9mbt" podStartSLOduration=17.184917085 podStartE2EDuration="24.783748984s" totalImagesPullingTime="7.598831899s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:51:31 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:51:55.783119328 +0000 UTC m=+62.863060302" watchObservedRunningTime="2026-07-09 16:51:55.783748984 +0000 UTC m=+62.863689954"
Jul 09 16:51:55.794578 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.794548    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj"]
Jul 09 16:51:55.797425 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:55.797401    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8"]
Jul 09 16:51:55.797615 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:55.797596    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6b0a740f_3ed3_4166_b8ec_aec444cb3ddb.slice/crio-7af62e86ba84c5b32d4dd468f3f2ae4c6fb83b7846137babc7687b9b6ca9763a WatchSource:0}: Error finding container 7af62e86ba84c5b32d4dd468f3f2ae4c6fb83b7846137babc7687b9b6ca9763a: Status 404 returned error can't find the container with id 7af62e86ba84c5b32d4dd468f3f2ae4c6fb83b7846137babc7687b9b6ca9763a
Jul 09 16:51:55.800601 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:51:55.800579    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd38a6b5d_4534_43d5_ac9a_4a951775eed5.slice/crio-df8dbc349d7b86898bf35d55319e20b3a040a7ad8836b17031b20c0c4d4f5cf1 WatchSource:0}: Error finding container df8dbc349d7b86898bf35d55319e20b3a040a7ad8836b17031b20c0c4d4f5cf1: Status 404 returned error can't find the container with id df8dbc349d7b86898bf35d55319e20b3a040a7ad8836b17031b20c0c4d4f5cf1
Jul 09 16:51:56.758469 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:56.758390    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj" event={"ID":"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb","Type":"ContainerStarted","Data":"7af62e86ba84c5b32d4dd468f3f2ae4c6fb83b7846137babc7687b9b6ca9763a"}
Jul 09 16:51:56.760994 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:56.760961    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8" event={"ID":"d38a6b5d-4534-43d5-ac9a-4a951775eed5","Type":"ContainerStarted","Data":"df8dbc349d7b86898bf35d55319e20b3a040a7ad8836b17031b20c0c4d4f5cf1"}
Jul 09 16:51:57.100199 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:57.100108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:57.159425 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:57.159343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:57.536912 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:57.536852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:57.598977 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:57.598942    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:57.933564 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:57.933521    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:57.986010 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:57.985922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:58.130183 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:58.130148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:58.348233 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:58.348119    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:58.543072 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:58.543036    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:58.594204 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:58.594135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:51:58.723256 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:58.723213    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-wgwn9"
Jul 09 16:51:59.171739 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.171690    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:59.172193 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.171750    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:59.172193 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.171803    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:59.174542 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.174519    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 16:51:59.175747 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.175724    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 16:51:59.175831 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.175747    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 16:51:59.185244 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.185217    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 16:51:59.185349 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.185222    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/ae1eeec8-a184-4829-b055-c7715ca287ff-original-pull-secret\") pod \"global-pull-secret-syncer-cf425\" (UID: \"ae1eeec8-a184-4829-b055-c7715ca287ff\") " pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:59.185349 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.185317    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dc814b9e-7680-4729-9858-c192c4de754c-metrics-certs\") pod \"network-metrics-daemon-qh9z6\" (UID: \"dc814b9e-7680-4729-9858-c192c4de754c\") " pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:59.196047 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.196019    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7rsh\" (UniqueName: \"kubernetes.io/projected/fb9c8550-2a74-4e96-8a11-42c778b6d711-kube-api-access-m7rsh\") pod \"network-check-target-fvr59\" (UID: \"fb9c8550-2a74-4e96-8a11-42c778b6d711\") " pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:59.261134 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.261095    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-cf425"
Jul 09 16:51:59.270187 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.270162    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-4tqb9\""
Jul 09 16:51:59.278288 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.278267    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh9z6"
Jul 09 16:51:59.363454 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.363426    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-2fgzg\""
Jul 09 16:51:59.371750 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.371722    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:51:59.902640 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:51:59.902589    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:00.001627 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.000545    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qh9z6"]
Jul 09 16:52:00.003130 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.003046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:00.010107 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:52:00.010077    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc814b9e_7680_4729_9858_c192c4de754c.slice/crio-aef44e578e5457d76b0530ac32c3912f9d4a7248abe68ae68acf59ecf65dae21 WatchSource:0}: Error finding container aef44e578e5457d76b0530ac32c3912f9d4a7248abe68ae68acf59ecf65dae21: Status 404 returned error can't find the container with id aef44e578e5457d76b0530ac32c3912f9d4a7248abe68ae68acf59ecf65dae21
Jul 09 16:52:00.230761 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.230734    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-cf425"]
Jul 09 16:52:00.233577 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:52:00.233429    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae1eeec8_a184_4829_b055_c7715ca287ff.slice/crio-19db2fe29cde52572927022894a927c31445c6e1ea05a9d6bb62a713d295b66a WatchSource:0}: Error finding container 19db2fe29cde52572927022894a927c31445c6e1ea05a9d6bb62a713d295b66a: Status 404 returned error can't find the container with id 19db2fe29cde52572927022894a927c31445c6e1ea05a9d6bb62a713d295b66a
Jul 09 16:52:00.233577 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.233573    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fvr59"]
Jul 09 16:52:00.235816 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:52:00.235789    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb9c8550_2a74_4e96_8a11_42c778b6d711.slice/crio-83ee74c926fac084fe45855d09a6a4d2bc1d7b249be50ecadee7eb3ee0045d7a WatchSource:0}: Error finding container 83ee74c926fac084fe45855d09a6a4d2bc1d7b249be50ecadee7eb3ee0045d7a: Status 404 returned error can't find the container with id 83ee74c926fac084fe45855d09a6a4d2bc1d7b249be50ecadee7eb3ee0045d7a
Jul 09 16:52:00.327194 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.327150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:00.428529 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.428496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:00.776118 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.776084    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fvr59" event={"ID":"fb9c8550-2a74-4e96-8a11-42c778b6d711","Type":"ContainerStarted","Data":"83ee74c926fac084fe45855d09a6a4d2bc1d7b249be50ecadee7eb3ee0045d7a"}
Jul 09 16:52:00.777397 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.777335    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qh9z6" event={"ID":"dc814b9e-7680-4729-9858-c192c4de754c","Type":"ContainerStarted","Data":"aef44e578e5457d76b0530ac32c3912f9d4a7248abe68ae68acf59ecf65dae21"}
Jul 09 16:52:00.779228 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.779199    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj" event={"ID":"6b0a740f-3ed3-4166-b8ec-aec444cb3ddb","Type":"ContainerStarted","Data":"78b8148d86ad051029325b083d8835df98f3974defb65cb6495bb8f7560c395a"}
Jul 09 16:52:00.782286 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.782260    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8" event={"ID":"d38a6b5d-4534-43d5-ac9a-4a951775eed5","Type":"ContainerStarted","Data":"caaaf365614a7124ed65a82eed636bd39d03db244c020b6ae57dfb1fb159f97a"}
Jul 09 16:52:00.782412 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.782302    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8" event={"ID":"d38a6b5d-4534-43d5-ac9a-4a951775eed5","Type":"ContainerStarted","Data":"8c7ca5f8a1c75d57c68d72a5735d1960c187f97864497e40c90227113511113e"}
Jul 09 16:52:00.782412 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.782322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8" event={"ID":"d38a6b5d-4534-43d5-ac9a-4a951775eed5","Type":"ContainerStarted","Data":"abf72bcddbc73ccb3b22248710271deafda28a7019bbda7e07e6c31f02cab29f"}
Jul 09 16:52:00.784458 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.784432    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cf425" event={"ID":"ae1eeec8-a184-4829-b055-c7715ca287ff","Type":"ContainerStarted","Data":"19db2fe29cde52572927022894a927c31445c6e1ea05a9d6bb62a713d295b66a"}
Jul 09 16:52:00.797991 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.797931    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-6484d57b46-x6txj" podStartSLOduration=2.739109671 podStartE2EDuration="6.797911435s" totalImagesPullingTime="4.058801764s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:00.797105353 +0000 UTC m=+67.877046323" watchObservedRunningTime="2026-07-09 16:52:00.797911435 +0000 UTC m=+67.877852406"
Jul 09 16:52:00.817984 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:00.817934    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7cbc58d5f7-hk7s8" podStartSLOduration=2.756570202 podStartE2EDuration="6.817915382s" totalImagesPullingTime="4.06134518s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:51:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:00.817295987 +0000 UTC m=+67.897236970" watchObservedRunningTime="2026-07-09 16:52:00.817915382 +0000 UTC m=+67.897856351"
Jul 09 16:52:01.799711 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:01.799673    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qh9z6" event={"ID":"dc814b9e-7680-4729-9858-c192c4de754c","Type":"ContainerStarted","Data":"304482e4c2abcc6cf0273023466a743730346bb70999a7f1020141bd30e3bdaa"}
Jul 09 16:52:02.742672 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:02.742585    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:02.806145 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:02.806108    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qh9z6" event={"ID":"dc814b9e-7680-4729-9858-c192c4de754c","Type":"ContainerStarted","Data":"a110b3895b85c17f94e604b665bc751676d9fc0173542d7034dadf073efb2776"}
Jul 09 16:52:02.825364 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:02.825242    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qh9z6" podStartSLOduration=68.169513937 podStartE2EDuration="1m9.825221381s" totalImagesPullingTime="1.655707444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:02.823511294 +0000 UTC m=+69.903452266" watchObservedRunningTime="2026-07-09 16:52:02.825221381 +0000 UTC m=+69.905162354"
Jul 09 16:52:02.832711 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:02.832646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:02.915213 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:02.915177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:03.015323 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:03.015188    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:03.079192 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:03.079150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:03.145456 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:03.145422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:03.760015 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:03.759978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:03.864503 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:03.864453    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:04.207075 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:04.207042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:04.248364 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:04.248329    2563 patch_prober.go:28] interesting pod/image-registry-64d7b4644d-ctptj container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 16:52:04.248544 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:04.248387    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj" podUID="b12220bf-d552-4c7f-b847-213503190981" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:52:04.307882 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:04.307838    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:05.801812 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:05.801781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:05.819840 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:05.819805    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-cf425" event={"ID":"ae1eeec8-a184-4829-b055-c7715ca287ff","Type":"ContainerStarted","Data":"d8561cfd6fb210c5c5c9bb846bc601a4b662109055c8c3ce1aa93cf1c6324f9f"}
Jul 09 16:52:05.821282 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:05.821258    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fvr59" event={"ID":"fb9c8550-2a74-4e96-8a11-42c778b6d711","Type":"ContainerStarted","Data":"b72ec92840f06a09e3270243a25e2815deee97adc09cf669fc0c1489dd2d5c4d"}
Jul 09 16:52:05.821395 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:05.821319    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:52:05.837894 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:05.837829    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-cf425" podStartSLOduration=67.835625906 podStartE2EDuration="1m12.837810307s" totalImagesPullingTime="5.002184401s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:05.835588138 +0000 UTC m=+72.915529110" watchObservedRunningTime="2026-07-09 16:52:05.837810307 +0000 UTC m=+72.917751278"
Jul 09 16:52:05.852201 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:05.852154    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fvr59" podStartSLOduration=67.858207985 podStartE2EDuration="1m12.852136551s" totalImagesPullingTime="4.993928566s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:05.851837978 +0000 UTC m=+72.931778947" watchObservedRunningTime="2026-07-09 16:52:05.852136551 +0000 UTC m=+72.932077521"
Jul 09 16:52:05.903120 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:05.903085    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:07.781338 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:07.781300    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:07.847092 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:07.847057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:07.945485 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:07.945446    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:08.046872 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:08.046790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:08.158104 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:08.158062    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:11.623048 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:11.623005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:11.723406 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:11.723365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:11.823857 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:11.823824    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:12.787996 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:12.787960    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:12.890725 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:12.890687    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:14.246099 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:14.246075    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:52:14.881383 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:14.881347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:14.983873 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:14.983746    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:15.081893 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:15.081845    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:16.178150 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:16.178112    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:16.243141 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:16.243111    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:16.765883 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:16.765856    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7f5469cd45-w4zzd"
Jul 09 16:52:16.868780 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:16.868744    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:16.968515 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:16.968471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:17.070496 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:17.070410    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:19.263985 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.263941    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj" podUID="b12220bf-d552-4c7f-b847-213503190981" containerName="registry" containerID="cri-o://1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728" gracePeriod=30
Jul 09 16:52:19.510009 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.509988    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:52:19.638352 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-installation-pull-secrets\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638541 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638359    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-registry-certificates\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638541 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638391    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b12220bf-d552-4c7f-b847-213503190981-ca-trust-extracted\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638541 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638412    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-image-registry-private-configuration\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638541 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638431    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-bound-sa-token\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638541 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638462    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638541 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638480    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-trusted-ca\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638541 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638505    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cxkvd\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-kube-api-access-cxkvd\") pod \"b12220bf-d552-4c7f-b847-213503190981\" (UID: \"b12220bf-d552-4c7f-b847-213503190981\") "
Jul 09 16:52:19.638913 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.638844    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:52:19.639274 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.639221    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:52:19.641039 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.640984    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:52:19.641393 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.641334    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:52:19.641393 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.641384    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:52:19.641515 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.641398    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:52:19.641515 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.641464    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-kube-api-access-cxkvd" (OuterVolumeSpecName: "kube-api-access-cxkvd") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "kube-api-access-cxkvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:52:19.647200 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.647180    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b12220bf-d552-4c7f-b847-213503190981-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "b12220bf-d552-4c7f-b847-213503190981" (UID: "b12220bf-d552-4c7f-b847-213503190981"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:52:19.739943 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739908    2563 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b12220bf-d552-4c7f-b847-213503190981-ca-trust-extracted\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.739943 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739939    2563 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-image-registry-private-configuration\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.739943 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739949    2563 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-bound-sa-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.740168 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739958    2563 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-registry-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.740168 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739967    2563 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-trusted-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.740168 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739975    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxkvd\" (UniqueName: \"kubernetes.io/projected/b12220bf-d552-4c7f-b847-213503190981-kube-api-access-cxkvd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.740168 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739984    2563 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b12220bf-d552-4c7f-b847-213503190981-installation-pull-secrets\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.740168 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.739992    2563 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b12220bf-d552-4c7f-b847-213503190981-registry-certificates\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:52:19.868054 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.868023    2563 generic.go:356] "Generic (PLEG): container finished" podID="b12220bf-d552-4c7f-b847-213503190981" containerID="1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728" exitCode=0
Jul 09 16:52:19.868212 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.868076    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj"
Jul 09 16:52:19.868212 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.868088    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj" event={"ID":"b12220bf-d552-4c7f-b847-213503190981","Type":"ContainerDied","Data":"1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728"}
Jul 09 16:52:19.868212 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.868128    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-64d7b4644d-ctptj" event={"ID":"b12220bf-d552-4c7f-b847-213503190981","Type":"ContainerDied","Data":"8be5660226a6b64c872f74bdd0889ace7893aa2fad4b7218a7a304a9a1219a13"}
Jul 09 16:52:19.868212 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.868148    2563 scope.go:117] "RemoveContainer" containerID="1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728"
Jul 09 16:52:19.876877 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.876860    2563 scope.go:117] "RemoveContainer" containerID="1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728"
Jul 09 16:52:19.877169 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:52:19.877148    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728\": container with ID starting with 1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728 not found: ID does not exist" containerID="1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728"
Jul 09 16:52:19.877223 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.877179    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728"} err="failed to get container status \"1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728\": rpc error: code = NotFound desc = could not find container \"1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728\": container with ID starting with 1f3db5cd2e5ba43e03aa77a268e6d75cfb5f88f442f18d5da03bf3b589725728 not found: ID does not exist"
Jul 09 16:52:19.889380 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.889327    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-64d7b4644d-ctptj"]
Jul 09 16:52:19.895084 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:19.895064    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-64d7b4644d-ctptj"]
Jul 09 16:52:21.351370 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:21.351340    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b12220bf-d552-4c7f-b847-213503190981" path="/var/lib/kubelet/pods/b12220bf-d552-4c7f-b847-213503190981/volumes"
Jul 09 16:52:22.969796 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:22.969760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:23.028699 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:23.028645    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:24.576186 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:24.576151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:25.518827 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:25.518772    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:25.598660 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:25.598623    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:25.860000 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:25.859961    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:25.915601 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:25.915564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:27.054324 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:27.054284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:27.203907 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:27.203873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.120805 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.120765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.222853 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.222809    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.387886 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.387798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.458979 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.458949    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.588201 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.588167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.648248 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.648164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.810333 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.810298    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:28.912195 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:28.912107    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:29.492266 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:29.492240    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:29.547740 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:29.547712    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:30.351834 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:30.351805    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:31.515581 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.515545    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-ch47v"]
Jul 09 16:52:31.516018 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.515878    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b12220bf-d552-4c7f-b847-213503190981" containerName="registry"
Jul 09 16:52:31.516018 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.515893    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b12220bf-d552-4c7f-b847-213503190981" containerName="registry"
Jul 09 16:52:31.516018 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.515953    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b12220bf-d552-4c7f-b847-213503190981" containerName="registry"
Jul 09 16:52:31.714055 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.714014    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.717698 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.717146    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 16:52:31.717698 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.717447    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 16:52:31.717698 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.717640    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-b9wdr\""
Jul 09 16:52:31.718940 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.718300    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 16:52:31.718940 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.718500    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 16:52:31.718940 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.718691    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 16:52:31.718940 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.718846    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 16:52:31.748252 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.748207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:31.822594 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822495    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-wtmp\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822594 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822544    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-textfile\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822594 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-metrics-client-ca\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822891 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822648    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmt8t\" (UniqueName: \"kubernetes.io/projected/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-kube-api-access-qmt8t\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822891 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822710    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822891 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822753    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-sys\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822891 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822790    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-tls\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822891 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822831    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-accelerators-collector-config\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.822891 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.822856    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-root\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.848479 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.848434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:31.923502 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923471    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-tls\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923502 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923506    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-accelerators-collector-config\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923535    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-root\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923580    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-wtmp\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-textfile\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:52:31.923621    2563 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923635    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-metrics-client-ca\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923672    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-root\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:52:31.923711    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-tls podName:270a2f78-b8fa-4ea2-bba9-fa3b9fed0741 nodeName:}" failed. No retries permitted until 2026-07-09 16:52:32.423689131 +0000 UTC m=+99.503630089 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-tls") pod "node-exporter-ch47v" (UID: "270a2f78-b8fa-4ea2-bba9-fa3b9fed0741") : secret "node-exporter-tls" not found
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923816    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-wtmp\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.923859 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923683    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmt8t\" (UniqueName: \"kubernetes.io/projected/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-kube-api-access-qmt8t\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.924285 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923897    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.924285 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.923945    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-sys\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.924285 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.924010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-sys\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.924285 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.924097    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-textfile\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.924285 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.924232    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-accelerators-collector-config\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.924508 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.924377    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-metrics-client-ca\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.926308 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.926284    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.937199 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.937169    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmt8t\" (UniqueName: \"kubernetes.io/projected/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-kube-api-access-qmt8t\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:31.950126 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:31.950087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:32.048975 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:32.048943    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:32.427646 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:32.427610    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-tls\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:32.430516 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:32.430488    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/270a2f78-b8fa-4ea2-bba9-fa3b9fed0741-node-exporter-tls\") pod \"node-exporter-ch47v\" (UID: \"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741\") " pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:32.628636 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:32.628589    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-ch47v"
Jul 09 16:52:32.638735 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:52:32.638708    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod270a2f78_b8fa_4ea2_bba9_fa3b9fed0741.slice/crio-72a29f39ca2e8b895bd1b80cae2802d904d8fbf90de77d08981885aeb15572f1 WatchSource:0}: Error finding container 72a29f39ca2e8b895bd1b80cae2802d904d8fbf90de77d08981885aeb15572f1: Status 404 returned error can't find the container with id 72a29f39ca2e8b895bd1b80cae2802d904d8fbf90de77d08981885aeb15572f1
Jul 09 16:52:32.765064 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:32.764980    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:32.818449 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:32.818415    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:32.924110 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:32.924074    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ch47v" event={"ID":"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741","Type":"ContainerStarted","Data":"72a29f39ca2e8b895bd1b80cae2802d904d8fbf90de77d08981885aeb15572f1"}
Jul 09 16:52:33.929458 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:33.929423    2563 generic.go:356] "Generic (PLEG): container finished" podID="270a2f78-b8fa-4ea2-bba9-fa3b9fed0741" containerID="640f173bf4a7647651cfe26c3c0a22fb3901cb0b1081dd3b17d58d8b952cda04" exitCode=0
Jul 09 16:52:33.929907 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:33.929487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ch47v" event={"ID":"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741","Type":"ContainerDied","Data":"640f173bf4a7647651cfe26c3c0a22fb3901cb0b1081dd3b17d58d8b952cda04"}
Jul 09 16:52:34.934451 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:34.934414    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ch47v" event={"ID":"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741","Type":"ContainerStarted","Data":"a5ad9ea104df7836db5acb26ae4214fdb3d0c7cac85a49ae728a5708b811fb89"}
Jul 09 16:52:34.934907 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:34.934474    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-ch47v" event={"ID":"270a2f78-b8fa-4ea2-bba9-fa3b9fed0741","Type":"ContainerStarted","Data":"0af9642b625f7d2d354a5b7a1073652fc8ed10a94660876b0ceb92fcc4b5a2a8"}
Jul 09 16:52:34.957540 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:34.957484    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-ch47v" podStartSLOduration=2.948722258 podStartE2EDuration="3.9574683s" totalImagesPullingTime="944.353411ms" totalInitContainerRuntime="64.392631ms" isStatefulPod=true podCreationTimestamp="2026-07-09 16:52:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:52:34.955713523 +0000 UTC m=+102.035654495" watchObservedRunningTime="2026-07-09 16:52:34.9574683 +0000 UTC m=+102.037409268"
Jul 09 16:52:35.119958 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:35.119929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:35.219800 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:35.219713    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:36.292283 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:36.292252    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:36.463846 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:36.463812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:36.568810 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:36.568725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:36.827374 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:36.827282    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fvr59"
Jul 09 16:52:37.652807 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:37.652770    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:45.327179 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:45.327146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:45.382695 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:45.382647    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:54.588008 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:54.587973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:54.641355 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:54.641321    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:55.739574 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:55.739540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:55.794551 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:55.794499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:52:55.845177 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:52:55.845144    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:53:21.767670 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:53:21.767620    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:53:31.816448 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:53:31.816416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:53:47.537776 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:53:47.537630    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:53:47.593670 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:53:47.593614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:53:56.966998 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:53:56.966955    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:53:57.017319 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:53:57.017283    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:53:59.513962 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:53:59.513926    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:00.030076 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:00.030042    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:08.075969 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:08.075927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:08.517202 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:08.517174    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:08.569117 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:08.569087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:15.335349 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:15.335317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:15.436722 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:15.436686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:32.043114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:32.043074    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:38.068257 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:38.068216    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:58.473477 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:58.473442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:54:58.612900 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:54:58.612870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:25.322556 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:25.322479    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:26.083539 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:26.083507    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:29.011967 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:29.011927    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:29.113987 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:29.113952    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:29.968982 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:29.968946    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:30.074515 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:30.074481    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:30.947086 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:30.947050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:31.042001 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:31.041964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:32.749571 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:32.749538    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:33.379850 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:33.379813    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:35.493802 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:35.493763    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:35.560534 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:35.560495    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:37.365114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:37.365071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:37.425949 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:37.425915    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:42.142252 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:42.142220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:42.781447 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:42.781414    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:44.927395 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:44.927360    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:49.205595 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:49.205565    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:49.462590 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:49.462509    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:51.163366 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:51.163330    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:55:53.258631 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:55:53.258606    2563 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 16:56:07.043263 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.043177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:07.234861 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.234828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:07.556382 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.556337    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:07.820705 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.820595    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:07.943084 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.943048    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:07.989963 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.989930    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"]
Jul 09 16:56:07.992992 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.992977    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:07.996085 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.996064    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 16:56:07.996691 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.996666    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-p2xv8\""
Jul 09 16:56:07.996820 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.996675    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 16:56:07.996954 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.996939    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 16:56:07.997105 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:07.997092    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 16:56:08.016301 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.016274    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"]
Jul 09 16:56:08.113640 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.113604    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ff4442-d40e-418f-8bf7-f3da82940523-tmp\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.114068 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.113673    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4ff4442-d40e-418f-8bf7-f3da82940523-manifests\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.114068 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.113705    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4ff4442-d40e-418f-8bf7-f3da82940523-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.114068 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.113723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7cc2\" (UniqueName: \"kubernetes.io/projected/b4ff4442-d40e-418f-8bf7-f3da82940523-kube-api-access-r7cc2\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.114068 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.113742    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4ff4442-d40e-418f-8bf7-f3da82940523-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.165638 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.165600    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:08.214969 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.214939    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4ff4442-d40e-418f-8bf7-f3da82940523-manifests\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.215129 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.214973    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4ff4442-d40e-418f-8bf7-f3da82940523-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.215129 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.214995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7cc2\" (UniqueName: \"kubernetes.io/projected/b4ff4442-d40e-418f-8bf7-f3da82940523-kube-api-access-r7cc2\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.215129 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.215015    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4ff4442-d40e-418f-8bf7-f3da82940523-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.215129 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.215059    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ff4442-d40e-418f-8bf7-f3da82940523-tmp\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.215426 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.215406    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/b4ff4442-d40e-418f-8bf7-f3da82940523-manifests\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.217440 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.217405    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b4ff4442-d40e-418f-8bf7-f3da82940523-tmp\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.217538 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.217481    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/b4ff4442-d40e-418f-8bf7-f3da82940523-apiservice-cert\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.217620 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.217602    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/b4ff4442-d40e-418f-8bf7-f3da82940523-webhook-cert\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.224413 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.224387    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7cc2\" (UniqueName: \"kubernetes.io/projected/b4ff4442-d40e-418f-8bf7-f3da82940523-kube-api-access-r7cc2\") pod \"opendatahub-operator-controller-manager-fdb769df8-6wzmg\" (UID: \"b4ff4442-d40e-418f-8bf7-f3da82940523\") " pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.264361 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.264325    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:08.302793 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.302759    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:08.366095 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.366010    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:08.437205 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.437174    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"]
Jul 09 16:56:08.440016 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.439988    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:08.440547 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:56:08.440526    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ff4442_d40e_418f_8bf7_f3da82940523.slice/crio-7e5c2c3081c18c906453403d57f7e96cbe3146b51eec32def5bce4c0c14bf427 WatchSource:0}: Error finding container 7e5c2c3081c18c906453403d57f7e96cbe3146b51eec32def5bce4c0c14bf427: Status 404 returned error can't find the container with id 7e5c2c3081c18c906453403d57f7e96cbe3146b51eec32def5bce4c0c14bf427
Jul 09 16:56:08.442322 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.442306    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 16:56:08.508840 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.508803    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:08.638679 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:08.638570    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg" event={"ID":"b4ff4442-d40e-418f-8bf7-f3da82940523","Type":"ContainerStarted","Data":"7e5c2c3081c18c906453403d57f7e96cbe3146b51eec32def5bce4c0c14bf427"}
Jul 09 16:56:11.655407 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:11.655369    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg" event={"ID":"b4ff4442-d40e-418f-8bf7-f3da82940523","Type":"ContainerStarted","Data":"969a23ad4ce78effe344f84db97d548fe50b0cfec8fb3b5d36077e2f1c9684dc"}
Jul 09 16:56:12.661215 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:12.661175    2563 generic.go:356] "Generic (PLEG): container finished" podID="b4ff4442-d40e-418f-8bf7-f3da82940523" containerID="969a23ad4ce78effe344f84db97d548fe50b0cfec8fb3b5d36077e2f1c9684dc" exitCode=0
Jul 09 16:56:12.661637 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:12.661253    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg" event={"ID":"b4ff4442-d40e-418f-8bf7-f3da82940523","Type":"ContainerDied","Data":"969a23ad4ce78effe344f84db97d548fe50b0cfec8fb3b5d36077e2f1c9684dc"}
Jul 09 16:56:13.667949 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:13.667913    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg" event={"ID":"b4ff4442-d40e-418f-8bf7-f3da82940523","Type":"ContainerStarted","Data":"cf9cd418c1c34802e8863f092236b87bb716489b7af6abc7ce6f940f9a65c7e9"}
Jul 09 16:56:13.668368 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:13.668037    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:13.690188 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:13.690143    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg" podStartSLOduration=2.559172287 podStartE2EDuration="6.690124272s" totalImagesPullingTime="3.105802176s" totalInitContainerRuntime="1.025149809s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:07 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:13.688031853 +0000 UTC m=+320.767972821" watchObservedRunningTime="2026-07-09 16:56:13.690124272 +0000 UTC m=+320.770065242"
Jul 09 16:56:15.815389 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:15.815351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:15.871196 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:15.871165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:19.999837 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:19.999799    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:20.119204 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:20.119146    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:24.674053 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:24.674021    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-fdb769df8-6wzmg"
Jul 09 16:56:27.711052 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:27.711016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:27.945782 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:27.945742    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:28.440945 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.440902    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:28.464881 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.464846    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"]
Jul 09 16:56:28.475700 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.475672    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.481099 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.481077    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 16:56:28.481273 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.481080    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 16:56:28.482252 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.482234    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-cfxsf\""
Jul 09 16:56:28.482396 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.482376    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 16:56:28.482460 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.482321    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 16:56:28.489207 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.489182    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"]
Jul 09 16:56:28.573194 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.573148    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7245h\" (UniqueName: \"kubernetes.io/projected/0c721106-f6c3-446e-bd65-82f745348116-kube-api-access-7245h\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.573194 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.573189    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c721106-f6c3-446e-bd65-82f745348116-tls-certs\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.573448 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.573216    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c721106-f6c3-446e-bd65-82f745348116-tmp\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.596635 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.596603    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8lnbc"]
Jul 09 16:56:28.599748 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.599730    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:28.602992 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.602970    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-24bc2\""
Jul 09 16:56:28.603207 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.603191    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 09 16:56:28.625241 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.625210    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8lnbc"]
Jul 09 16:56:28.674492 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.674459    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c721106-f6c3-446e-bd65-82f745348116-tmp\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.674698 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.674639    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7245h\" (UniqueName: \"kubernetes.io/projected/0c721106-f6c3-446e-bd65-82f745348116-kube-api-access-7245h\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.674774 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.674713    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c721106-f6c3-446e-bd65-82f745348116-tls-certs\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.674862 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:56:28.674840    2563 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 09 16:56:28.674997 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:56:28.674916    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0c721106-f6c3-446e-bd65-82f745348116-tls-certs podName:0c721106-f6c3-446e-bd65-82f745348116 nodeName:}" failed. No retries permitted until 2026-07-09 16:56:29.174896761 +0000 UTC m=+336.254837723 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/0c721106-f6c3-446e-bd65-82f745348116-tls-certs") pod "kube-auth-proxy-6b94bd4c99-6lpfl" (UID: "0c721106-f6c3-446e-bd65-82f745348116") : secret "kube-auth-proxy-tls" not found
Jul 09 16:56:28.677030 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.677010    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0c721106-f6c3-446e-bd65-82f745348116-tmp\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.689562 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.689539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7245h\" (UniqueName: \"kubernetes.io/projected/0c721106-f6c3-446e-bd65-82f745348116-kube-api-access-7245h\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:28.775456 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.775365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhws9\" (UniqueName: \"kubernetes.io/projected/06bb29c5-4dfd-42cc-944a-563d29c04fc5-kube-api-access-rhws9\") pod \"odh-model-controller-7998c67dc4-8lnbc\" (UID: \"06bb29c5-4dfd-42cc-944a-563d29c04fc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:28.775456 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.775433    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06bb29c5-4dfd-42cc-944a-563d29c04fc5-cert\") pod \"odh-model-controller-7998c67dc4-8lnbc\" (UID: \"06bb29c5-4dfd-42cc-944a-563d29c04fc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:28.876762 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.876724    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhws9\" (UniqueName: \"kubernetes.io/projected/06bb29c5-4dfd-42cc-944a-563d29c04fc5-kube-api-access-rhws9\") pod \"odh-model-controller-7998c67dc4-8lnbc\" (UID: \"06bb29c5-4dfd-42cc-944a-563d29c04fc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:28.876762 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.876766    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06bb29c5-4dfd-42cc-944a-563d29c04fc5-cert\") pod \"odh-model-controller-7998c67dc4-8lnbc\" (UID: \"06bb29c5-4dfd-42cc-944a-563d29c04fc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:28.877017 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:56:28.876894    2563 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 16:56:28.877017 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:56:28.876941    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06bb29c5-4dfd-42cc-944a-563d29c04fc5-cert podName:06bb29c5-4dfd-42cc-944a-563d29c04fc5 nodeName:}" failed. No retries permitted until 2026-07-09 16:56:29.376928781 +0000 UTC m=+336.456869729 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/06bb29c5-4dfd-42cc-944a-563d29c04fc5-cert") pod "odh-model-controller-7998c67dc4-8lnbc" (UID: "06bb29c5-4dfd-42cc-944a-563d29c04fc5") : secret "odh-model-controller-webhook-cert" not found
Jul 09 16:56:28.896604 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:28.896552    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhws9\" (UniqueName: \"kubernetes.io/projected/06bb29c5-4dfd-42cc-944a-563d29c04fc5-kube-api-access-rhws9\") pod \"odh-model-controller-7998c67dc4-8lnbc\" (UID: \"06bb29c5-4dfd-42cc-944a-563d29c04fc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:29.021239 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.021195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:29.115041 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.115009    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:29.179398 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.179364    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c721106-f6c3-446e-bd65-82f745348116-tls-certs\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:29.181844 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.181818    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c721106-f6c3-446e-bd65-82f745348116-tls-certs\") pod \"kube-auth-proxy-6b94bd4c99-6lpfl\" (UID: \"0c721106-f6c3-446e-bd65-82f745348116\") " pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:29.380767 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.380668    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06bb29c5-4dfd-42cc-944a-563d29c04fc5-cert\") pod \"odh-model-controller-7998c67dc4-8lnbc\" (UID: \"06bb29c5-4dfd-42cc-944a-563d29c04fc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:29.383114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.383088    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/06bb29c5-4dfd-42cc-944a-563d29c04fc5-cert\") pod \"odh-model-controller-7998c67dc4-8lnbc\" (UID: \"06bb29c5-4dfd-42cc-944a-563d29c04fc5\") " pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:29.386035 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.386016    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"
Jul 09 16:56:29.510888 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.510856    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:29.542112 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.541782    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl"]
Jul 09 16:56:29.544708 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:56:29.544670    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c721106_f6c3_446e_bd65_82f745348116.slice/crio-7a0c146c95d1d943c31cafbc8830be9454efcf405e8c4a8c2d2b8fe7e23cac82 WatchSource:0}: Error finding container 7a0c146c95d1d943c31cafbc8830be9454efcf405e8c4a8c2d2b8fe7e23cac82: Status 404 returned error can't find the container with id 7a0c146c95d1d943c31cafbc8830be9454efcf405e8c4a8c2d2b8fe7e23cac82
Jul 09 16:56:29.659887 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.659801    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:29.690401 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.690337    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-8lnbc"]
Jul 09 16:56:29.692954 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:56:29.692927    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06bb29c5_4dfd_42cc_944a_563d29c04fc5.slice/crio-6b2f76d185e23a350e78c0bc0818b8906f579dd0d9f5afebd36e23adbb258915 WatchSource:0}: Error finding container 6b2f76d185e23a350e78c0bc0818b8906f579dd0d9f5afebd36e23adbb258915: Status 404 returned error can't find the container with id 6b2f76d185e23a350e78c0bc0818b8906f579dd0d9f5afebd36e23adbb258915
Jul 09 16:56:29.723521 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.723462    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl" event={"ID":"0c721106-f6c3-446e-bd65-82f745348116","Type":"ContainerStarted","Data":"7a0c146c95d1d943c31cafbc8830be9454efcf405e8c4a8c2d2b8fe7e23cac82"}
Jul 09 16:56:29.724978 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.724942    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc" event={"ID":"06bb29c5-4dfd-42cc-944a-563d29c04fc5","Type":"ContainerStarted","Data":"6b2f76d185e23a350e78c0bc0818b8906f579dd0d9f5afebd36e23adbb258915"}
Jul 09 16:56:29.895629 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:29.895586    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:30.506142 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.505950    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-42t58"]
Jul 09 16:56:30.510064 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.509320    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.514314 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.513359    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 16:56:30.514314 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.513664    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-4fxxr\""
Jul 09 16:56:30.514314 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.513959    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 16:56:30.514314 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.514119    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 16:56:30.514314 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.514161    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 16:56:30.540379 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.540271    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-42t58"]
Jul 09 16:56:30.662344 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.662293    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:30.691076 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.690971    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-cacerts\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.691076 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.691013    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.691321 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.691083    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908d3133-152d-4469-b33c-d9afb4e2e565-local-certs\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.691321 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.691166    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-istio-token\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.691321 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.691193    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.691321 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.691234    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7bnsd\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-kube-api-access-7bnsd\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.691321 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.691318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.732506 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.732299    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:30.792166 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.792075    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-cacerts\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.792166 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.792123    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.792166 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.792150    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908d3133-152d-4469-b33c-d9afb4e2e565-local-certs\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.792491 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.792355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-istio-token\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.792491 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.792393    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.792491 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.792423    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7bnsd\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-kube-api-access-7bnsd\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.792632 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.792497    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.793897 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.793849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.797678 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.797617    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.799461 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.799436    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908d3133-152d-4469-b33c-d9afb4e2e565-local-certs\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.799680 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.799559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-cacerts\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.799805 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.799765    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.816349 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.816265    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7bnsd\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-kube-api-access-7bnsd\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.819894 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.819864    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-istio-token\") pod \"istiod-openshift-gateway-7574459656-42t58\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.825843 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.825623    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:30.997926 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:30.997893    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-42t58"]
Jul 09 16:56:31.002237 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:56:31.002204    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod908d3133_152d_4469_b33c_d9afb4e2e565.slice/crio-ae1adce97892a679db35a11f2c0cfe79ffe9058b9a9959c788ee3419062a9a57 WatchSource:0}: Error finding container ae1adce97892a679db35a11f2c0cfe79ffe9058b9a9959c788ee3419062a9a57: Status 404 returned error can't find the container with id ae1adce97892a679db35a11f2c0cfe79ffe9058b9a9959c788ee3419062a9a57
Jul 09 16:56:31.739114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:31.739077    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58" event={"ID":"908d3133-152d-4469-b33c-d9afb4e2e565","Type":"ContainerStarted","Data":"ae1adce97892a679db35a11f2c0cfe79ffe9058b9a9959c788ee3419062a9a57"}
Jul 09 16:56:31.935081 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:31.934984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:34.918131 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:34.918089    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 16:56:34.918454 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:34.918172    2563 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236224Ki","pods":"250"}
Jul 09 16:56:35.416497 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.416447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:35.516431 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.516394    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:35.756500 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.756413    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc" event={"ID":"06bb29c5-4dfd-42cc-944a-563d29c04fc5","Type":"ContainerStarted","Data":"91f61c674f65d73b127b11bf1a1fa1c18fc84eb83f2bf80ab8cfc9a09305f588"}
Jul 09 16:56:35.756680 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.756538    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:35.758113 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.758089    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58" event={"ID":"908d3133-152d-4469-b33c-d9afb4e2e565","Type":"ContainerStarted","Data":"e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d"}
Jul 09 16:56:35.758238 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.758192    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:35.759597 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.759577    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl" event={"ID":"0c721106-f6c3-446e-bd65-82f745348116","Type":"ContainerStarted","Data":"05d70a8b31bca7a2e1f8c71cd76c5fd7716bfa2fba913b940e1da6c0c962abf9"}
Jul 09 16:56:35.760173 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.760155    2563 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-42t58 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 16:56:35.760229 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.760208    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58" podUID="908d3133-152d-4469-b33c-d9afb4e2e565" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 16:56:35.785647 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.785584    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc" podStartSLOduration=2.576687214 podStartE2EDuration="7.785555589s" totalImagesPullingTime="5.208868375s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:35.784245749 +0000 UTC m=+342.864186720" watchObservedRunningTime="2026-07-09 16:56:35.785555589 +0000 UTC m=+342.865496559"
Jul 09 16:56:35.814060 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.814004    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b94bd4c99-6lpfl" podStartSLOduration=2.443451107 podStartE2EDuration="7.813988523s" totalImagesPullingTime="5.370537416s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:35.812560823 +0000 UTC m=+342.892501793" watchObservedRunningTime="2026-07-09 16:56:35.813988523 +0000 UTC m=+342.893929525"
Jul 09 16:56:35.841398 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:35.841341    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58" podStartSLOduration=1.930302011 podStartE2EDuration="5.841319343s" totalImagesPullingTime="3.911017332s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:35.840486257 +0000 UTC m=+342.920427237" watchObservedRunningTime="2026-07-09 16:56:35.841319343 +0000 UTC m=+342.921260313"
Jul 09 16:56:36.347236 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:36.347194    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:36.458050 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:36.457998    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:36.763695 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:36.763664    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:36.832611 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:36.832578    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:39.128079 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.128049    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"]
Jul 09 16:56:39.131422 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.131404    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.135400 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.135365    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 09 16:56:39.135400 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.135367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 09 16:56:39.135584 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.135407    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 09 16:56:39.135584 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.135367    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 09 16:56:39.135584 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.135437    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 09 16:56:39.135584 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.135368    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-nv68j\""
Jul 09 16:56:39.141868 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.141842    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"]
Jul 09 16:56:39.155819 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.155794    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4fqsn\" (UniqueName: \"kubernetes.io/projected/d6a3ad51-e423-4f2f-818b-fece3dd0c742-kube-api-access-4fqsn\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.155935 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.155848    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6a3ad51-e423-4f2f-818b-fece3dd0c742-cert\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.155935 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.155879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6a3ad51-e423-4f2f-818b-fece3dd0c742-manager-config\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.155935 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.155908    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6a3ad51-e423-4f2f-818b-fece3dd0c742-metrics-cert\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.256313 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.256276    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4fqsn\" (UniqueName: \"kubernetes.io/projected/d6a3ad51-e423-4f2f-818b-fece3dd0c742-kube-api-access-4fqsn\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.256514 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.256328    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6a3ad51-e423-4f2f-818b-fece3dd0c742-cert\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.256514 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.256353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6a3ad51-e423-4f2f-818b-fece3dd0c742-manager-config\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.256514 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.256378    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6a3ad51-e423-4f2f-818b-fece3dd0c742-metrics-cert\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.257120 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.257094    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/d6a3ad51-e423-4f2f-818b-fece3dd0c742-manager-config\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.258789 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.258766    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/d6a3ad51-e423-4f2f-818b-fece3dd0c742-metrics-cert\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.258890 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.258854    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6a3ad51-e423-4f2f-818b-fece3dd0c742-cert\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.272239 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.272210    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4fqsn\" (UniqueName: \"kubernetes.io/projected/d6a3ad51-e423-4f2f-818b-fece3dd0c742-kube-api-access-4fqsn\") pod \"lws-controller-manager-5c969b7c4d-ft6dw\" (UID: \"d6a3ad51-e423-4f2f-818b-fece3dd0c742\") " pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.440610 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.440519    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:39.569337 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.569197    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"]
Jul 09 16:56:39.571481 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:56:39.571454    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a3ad51_e423_4f2f_818b_fece3dd0c742.slice/crio-0684faa7318625eb758e299b8eccbe85eaeb89b327d577dc1b2fd20fad6d88c2 WatchSource:0}: Error finding container 0684faa7318625eb758e299b8eccbe85eaeb89b327d577dc1b2fd20fad6d88c2: Status 404 returned error can't find the container with id 0684faa7318625eb758e299b8eccbe85eaeb89b327d577dc1b2fd20fad6d88c2
Jul 09 16:56:39.774944 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:39.774863    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw" event={"ID":"d6a3ad51-e423-4f2f-818b-fece3dd0c742","Type":"ContainerStarted","Data":"0684faa7318625eb758e299b8eccbe85eaeb89b327d577dc1b2fd20fad6d88c2"}
Jul 09 16:56:40.512539 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:40.512317    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:40.639373 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:40.639276    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:40.726882 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:40.726836    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:41.337642 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:41.337610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:41.834100 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:41.834050    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:41.935057 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:41.935021    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:42.056712 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:42.056678    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:56:42.787465 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:42.787428    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw" event={"ID":"d6a3ad51-e423-4f2f-818b-fece3dd0c742","Type":"ContainerStarted","Data":"7f211b61f1887b6474653714736fe8d6a3f340e742a4f69d92928f36b9f257d1"}
Jul 09 16:56:42.787643 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:42.787559    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:56:42.826263 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:42.826213    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw" podStartSLOduration=1.465354939 podStartE2EDuration="3.82619551s" totalImagesPullingTime="2.360840571s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:56:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:56:42.824978331 +0000 UTC m=+349.904919301" watchObservedRunningTime="2026-07-09 16:56:42.82619551 +0000 UTC m=+349.906136460"
Jul 09 16:56:43.911388 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:43.911355    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-42t58"]
Jul 09 16:56:43.911847 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:43.911580    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58" podUID="908d3133-152d-4469-b33c-d9afb4e2e565" containerName="discovery" containerID="cri-o://e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d" gracePeriod=30
Jul 09 16:56:44.163317 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.163258    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:44.203397 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.203371    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-istio-token\") pod \"908d3133-152d-4469-b33c-d9afb4e2e565\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") "
Jul 09 16:56:44.203630 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.203405    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-cacerts\") pod \"908d3133-152d-4469-b33c-d9afb4e2e565\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") "
Jul 09 16:56:44.203630 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.203460    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-dns-cert\") pod \"908d3133-152d-4469-b33c-d9afb4e2e565\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") "
Jul 09 16:56:44.203630 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.203505    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908d3133-152d-4469-b33c-d9afb4e2e565-local-certs\") pod \"908d3133-152d-4469-b33c-d9afb4e2e565\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") "
Jul 09 16:56:44.203630 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.203523    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-kubeconfig\") pod \"908d3133-152d-4469-b33c-d9afb4e2e565\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") "
Jul 09 16:56:44.203630 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.203554    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-ca-configmap\") pod \"908d3133-152d-4469-b33c-d9afb4e2e565\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") "
Jul 09 16:56:44.203630 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.203595    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7bnsd\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-kube-api-access-7bnsd\") pod \"908d3133-152d-4469-b33c-d9afb4e2e565\" (UID: \"908d3133-152d-4469-b33c-d9afb4e2e565\") "
Jul 09 16:56:44.204192 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.204141    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "908d3133-152d-4469-b33c-d9afb4e2e565" (UID: "908d3133-152d-4469-b33c-d9afb4e2e565"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 16:56:44.206076 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.206043    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "908d3133-152d-4469-b33c-d9afb4e2e565" (UID: "908d3133-152d-4469-b33c-d9afb4e2e565"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:56:44.206076 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.206053    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-istio-token" (OuterVolumeSpecName: "istio-token") pod "908d3133-152d-4469-b33c-d9afb4e2e565" (UID: "908d3133-152d-4469-b33c-d9afb4e2e565"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:56:44.206213 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.206075    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-cacerts" (OuterVolumeSpecName: "cacerts") pod "908d3133-152d-4469-b33c-d9afb4e2e565" (UID: "908d3133-152d-4469-b33c-d9afb4e2e565"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:56:44.206213 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.206171    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "908d3133-152d-4469-b33c-d9afb4e2e565" (UID: "908d3133-152d-4469-b33c-d9afb4e2e565"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:56:44.206213 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.206185    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d3133-152d-4469-b33c-d9afb4e2e565-local-certs" (OuterVolumeSpecName: "local-certs") pod "908d3133-152d-4469-b33c-d9afb4e2e565" (UID: "908d3133-152d-4469-b33c-d9afb4e2e565"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 16:56:44.206329 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.206259    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-kube-api-access-7bnsd" (OuterVolumeSpecName: "kube-api-access-7bnsd") pod "908d3133-152d-4469-b33c-d9afb4e2e565" (UID: "908d3133-152d-4469-b33c-d9afb4e2e565"). InnerVolumeSpecName "kube-api-access-7bnsd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:56:44.305008 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.304972    2563 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/908d3133-152d-4469-b33c-d9afb4e2e565-local-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:56:44.305008 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.304997    2563 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-kubeconfig\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:56:44.305008 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.305006    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-ca-configmap\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:56:44.305228 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.305017    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7bnsd\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-kube-api-access-7bnsd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:56:44.305228 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.305028    2563 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/908d3133-152d-4469-b33c-d9afb4e2e565-istio-token\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:56:44.305228 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.305036    2563 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-cacerts\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:56:44.305228 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.305044    2563 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/908d3133-152d-4469-b33c-d9afb4e2e565-istio-csr-dns-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:56:44.795664 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.795631    2563 generic.go:356] "Generic (PLEG): container finished" podID="908d3133-152d-4469-b33c-d9afb4e2e565" containerID="e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d" exitCode=0
Jul 09 16:56:44.795838 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.795699    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58"
Jul 09 16:56:44.795838 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.795702    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58" event={"ID":"908d3133-152d-4469-b33c-d9afb4e2e565","Type":"ContainerDied","Data":"e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d"}
Jul 09 16:56:44.795838 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.795806    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-42t58" event={"ID":"908d3133-152d-4469-b33c-d9afb4e2e565","Type":"ContainerDied","Data":"ae1adce97892a679db35a11f2c0cfe79ffe9058b9a9959c788ee3419062a9a57"}
Jul 09 16:56:44.795838 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.795824    2563 scope.go:117] "RemoveContainer" containerID="e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d"
Jul 09 16:56:44.804352 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.804334    2563 scope.go:117] "RemoveContainer" containerID="e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d"
Jul 09 16:56:44.804608 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:56:44.804590    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d\": container with ID starting with e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d not found: ID does not exist" containerID="e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d"
Jul 09 16:56:44.804676 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.804616    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d"} err="failed to get container status \"e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d\": rpc error: code = NotFound desc = could not find container \"e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d\": container with ID starting with e87b30c33d0cc0dce46c872d617650da76265315c3e916aec6604a1515498b6d not found: ID does not exist"
Jul 09 16:56:44.815463 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.815439    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-42t58"]
Jul 09 16:56:44.821270 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:44.821248    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-42t58"]
Jul 09 16:56:45.352542 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:45.352510    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908d3133-152d-4469-b33c-d9afb4e2e565" path="/var/lib/kubelet/pods/908d3133-152d-4469-b33c-d9afb4e2e565/volumes"
Jul 09 16:56:46.765913 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:46.765882    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-8lnbc"
Jul 09 16:56:53.792603 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:56:53.792573    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5c969b7c4d-ft6dw"
Jul 09 16:57:29.136190 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:57:29.136150    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:57:29.236884 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:57:29.236847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:57:29.422159 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:57:29.422064    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:57:53.284916 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:57:53.284884    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:57:53.421120 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:57:53.421083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:57:54.020705 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:57:54.020668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:04.280796 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:04.280756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:04.452187 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:04.452142    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:04.539500 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:04.539404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:04.688203 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:04.688164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:07.576355 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:07.576309    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:09.742976 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:09.742936    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:10.084962 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:10.084877    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:10.751479 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:10.751444    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:10.878802 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:10.878760    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:12.778434 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:12.778399    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:14.689520 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:14.689488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:14.952338 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:14.952242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:17.342921 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:17.342886    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:18.857995 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.857964    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"]
Jul 09 16:58:18.858369 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.858265    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908d3133-152d-4469-b33c-d9afb4e2e565" containerName="discovery"
Jul 09 16:58:18.858369 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.858277    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="908d3133-152d-4469-b33c-d9afb4e2e565" containerName="discovery"
Jul 09 16:58:18.858369 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.858321    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="908d3133-152d-4469-b33c-d9afb4e2e565" containerName="discovery"
Jul 09 16:58:18.861152 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.861131    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:18.864238 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.864220    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 09 16:58:18.864342 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.864223    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 16:58:18.864342 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.864300    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-47zsg\""
Jul 09 16:58:18.864451 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.864245    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 09 16:58:18.865382 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.865368    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 16:58:18.886486 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.886464    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"]
Jul 09 16:58:18.900475 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.900443    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:18.903422 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.903404    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48a6abca-cf18-4339-9831-e5e7217c34fb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:18.903499 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.903430    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n798w\" (UniqueName: \"kubernetes.io/projected/48a6abca-cf18-4339-9831-e5e7217c34fb-kube-api-access-n798w\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:18.903499 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:18.903467    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/48a6abca-cf18-4339-9831-e5e7217c34fb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.004114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.004078    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48a6abca-cf18-4339-9831-e5e7217c34fb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.004114 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.004115    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n798w\" (UniqueName: \"kubernetes.io/projected/48a6abca-cf18-4339-9831-e5e7217c34fb-kube-api-access-n798w\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.004316 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.004176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/48a6abca-cf18-4339-9831-e5e7217c34fb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.004877 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.004860    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/48a6abca-cf18-4339-9831-e5e7217c34fb-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.006645 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.006627    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/48a6abca-cf18-4339-9831-e5e7217c34fb-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.013202 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.013178    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n798w\" (UniqueName: \"kubernetes.io/projected/48a6abca-cf18-4339-9831-e5e7217c34fb-kube-api-access-n798w\") pod \"kuadrant-console-plugin-66b5774985-bpjgk\" (UID: \"48a6abca-cf18-4339-9831-e5e7217c34fb\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.022523 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.022496    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:19.170516 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.170438    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"
Jul 09 16:58:19.254152 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.254118    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:19.298601 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.298580    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk"]
Jul 09 16:58:19.301153 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:58:19.301128    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod48a6abca_cf18_4339_9831_e5e7217c34fb.slice/crio-46d08f7ddcabb116a31838fbd52ae6deabbd6b3278afc28a785b0c9aa8d6d0c9 WatchSource:0}: Error finding container 46d08f7ddcabb116a31838fbd52ae6deabbd6b3278afc28a785b0c9aa8d6d0c9: Status 404 returned error can't find the container with id 46d08f7ddcabb116a31838fbd52ae6deabbd6b3278afc28a785b0c9aa8d6d0c9
Jul 09 16:58:19.349298 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:19.349270    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:20.135395 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:20.135360    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk" event={"ID":"48a6abca-cf18-4339-9831-e5e7217c34fb","Type":"ContainerStarted","Data":"46d08f7ddcabb116a31838fbd52ae6deabbd6b3278afc28a785b0c9aa8d6d0c9"}
Jul 09 16:58:22.153981 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:22.153941    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:28.161197 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:28.161159    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:28.283964 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:28.283929    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:28.387344 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:28.387296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:30.391586 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:30.391547    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:30.499783 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:30.499747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:35.104138 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:35.104105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:35.207439 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:35.207406    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:38.910223 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:38.910181    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:39.008241 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:39.008187    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:41.228663 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:41.228625    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk" event={"ID":"48a6abca-cf18-4339-9831-e5e7217c34fb","Type":"ContainerStarted","Data":"35881404b430b23a25cc33c87e8f6d36c048a1d046a7b766e3888f59e4ab9429"}
Jul 09 16:58:41.246778 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:41.246728    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-bpjgk" podStartSLOduration=1.809888701 podStartE2EDuration="23.246713192s" totalImagesPullingTime="21.436824491s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:58:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:58:41.246600459 +0000 UTC m=+468.326541428" watchObservedRunningTime="2026-07-09 16:58:41.246713192 +0000 UTC m=+468.326654161"
Jul 09 16:58:47.711022 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:47.710984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:47.811094 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:47.811061    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:47.913093 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:47.913059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:48.380321 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:48.380287    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:50.141587 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:50.141557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:50.317939 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:50.317903    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:54.894617 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:54.894587    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:54.996234 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:54.996201    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:56.867572 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:56.867540    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:56.922184 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:56.922148    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:59.112576 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:59.112541    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:58:59.173861 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:58:59.173828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:01.836699 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:01.836668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:02.065290 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:02.065256    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:06.365526 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:06.365492    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:06.485628 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:06.485596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:06.585939 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:06.585905    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:07.163180 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.163149    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-l5fjt"]
Jul 09 16:59:07.200775 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.200728    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:07.216937 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.216912    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5fjt"]
Jul 09 16:59:07.217067 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.216991    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5fjt"
Jul 09 16:59:07.219802 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.219779    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l6v8p\""
Jul 09 16:59:07.277730 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.277696    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-vsxtg"]
Jul 09 16:59:07.280853 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.280837    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vsxtg"
Jul 09 16:59:07.284546 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.284527    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vsxtg"]
Jul 09 16:59:07.305377 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.305345    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qs67p\" (UniqueName: \"kubernetes.io/projected/1b8b856f-0daf-47b1-aa53-c0a87cfffbb0-kube-api-access-qs67p\") pod \"authorino-75d556bd77-vsxtg\" (UID: \"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0\") " pod="kuadrant-system/authorino-75d556bd77-vsxtg"
Jul 09 16:59:07.351773 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.351745    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:07.405764 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.405731    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qs67p\" (UniqueName: \"kubernetes.io/projected/1b8b856f-0daf-47b1-aa53-c0a87cfffbb0-kube-api-access-qs67p\") pod \"authorino-75d556bd77-vsxtg\" (UID: \"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0\") " pod="kuadrant-system/authorino-75d556bd77-vsxtg"
Jul 09 16:59:07.406140 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.405796    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58mfh\" (UniqueName: \"kubernetes.io/projected/213b86b6-9d1a-4e4a-8840-f84e08dc496c-kube-api-access-58mfh\") pod \"authorino-58cd955675-l5fjt\" (UID: \"213b86b6-9d1a-4e4a-8840-f84e08dc496c\") " pod="kuadrant-system/authorino-58cd955675-l5fjt"
Jul 09 16:59:07.414426 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.414358    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qs67p\" (UniqueName: \"kubernetes.io/projected/1b8b856f-0daf-47b1-aa53-c0a87cfffbb0-kube-api-access-qs67p\") pod \"authorino-75d556bd77-vsxtg\" (UID: \"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0\") " pod="kuadrant-system/authorino-75d556bd77-vsxtg"
Jul 09 16:59:07.507026 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.506996    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58mfh\" (UniqueName: \"kubernetes.io/projected/213b86b6-9d1a-4e4a-8840-f84e08dc496c-kube-api-access-58mfh\") pod \"authorino-58cd955675-l5fjt\" (UID: \"213b86b6-9d1a-4e4a-8840-f84e08dc496c\") " pod="kuadrant-system/authorino-58cd955675-l5fjt"
Jul 09 16:59:07.515558 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.515531    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58mfh\" (UniqueName: \"kubernetes.io/projected/213b86b6-9d1a-4e4a-8840-f84e08dc496c-kube-api-access-58mfh\") pod \"authorino-58cd955675-l5fjt\" (UID: \"213b86b6-9d1a-4e4a-8840-f84e08dc496c\") " pod="kuadrant-system/authorino-58cd955675-l5fjt"
Jul 09 16:59:07.526325 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.526301    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5fjt"
Jul 09 16:59:07.590545 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.590513    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vsxtg"
Jul 09 16:59:07.649954 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.649599    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5fjt"]
Jul 09 16:59:07.721150 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:07.721114    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vsxtg"]
Jul 09 16:59:07.724074 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:59:07.724047    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1b8b856f_0daf_47b1_aa53_c0a87cfffbb0.slice/crio-48c6d3de89b8a885ca7d8c33519227c4c22d471932e20f879104d039535a349b WatchSource:0}: Error finding container 48c6d3de89b8a885ca7d8c33519227c4c22d471932e20f879104d039535a349b: Status 404 returned error can't find the container with id 48c6d3de89b8a885ca7d8c33519227c4c22d471932e20f879104d039535a349b
Jul 09 16:59:08.116957 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:08.116922    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:08.221020 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:08.220958    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:08.332957 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:08.332916    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5fjt" event={"ID":"213b86b6-9d1a-4e4a-8840-f84e08dc496c","Type":"ContainerStarted","Data":"784071bfb9112b24c2e7d4f47f5c655494a2de0bf55330c489ec7fa00ce03033"}
Jul 09 16:59:08.339189 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:08.339134    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vsxtg" event={"ID":"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0","Type":"ContainerStarted","Data":"48c6d3de89b8a885ca7d8c33519227c4c22d471932e20f879104d039535a349b"}
Jul 09 16:59:09.110425 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:09.110281    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:09.212455 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:09.212265    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:10.572586 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:10.572552    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:10.632209 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:10.632158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:11.353012 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:11.352905    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vsxtg" event={"ID":"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0","Type":"ContainerStarted","Data":"67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996"}
Jul 09 16:59:11.354183 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:11.354159    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5fjt" event={"ID":"213b86b6-9d1a-4e4a-8840-f84e08dc496c","Type":"ContainerStarted","Data":"81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145"}
Jul 09 16:59:11.367472 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:11.367429    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-vsxtg" podStartSLOduration=1.063440164 podStartE2EDuration="4.367414168s" totalImagesPullingTime="3.303974004s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:59:11.366453863 +0000 UTC m=+498.446394843" watchObservedRunningTime="2026-07-09 16:59:11.367414168 +0000 UTC m=+498.447355139"
Jul 09 16:59:11.380737 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:11.380688    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-l5fjt" podStartSLOduration=0.996487282 podStartE2EDuration="4.380671514s" totalImagesPullingTime="3.384184232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:59:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:59:11.379706445 +0000 UTC m=+498.459647426" watchObservedRunningTime="2026-07-09 16:59:11.380671514 +0000 UTC m=+498.460612477"
Jul 09 16:59:11.404494 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:11.404460    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5fjt"]
Jul 09 16:59:13.361843 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:13.361803    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-l5fjt" podUID="213b86b6-9d1a-4e4a-8840-f84e08dc496c" containerName="authorino" containerID="cri-o://81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145" gracePeriod=30
Jul 09 16:59:13.597681 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:13.597645    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5fjt"
Jul 09 16:59:13.661744 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:13.661637    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58mfh\" (UniqueName: \"kubernetes.io/projected/213b86b6-9d1a-4e4a-8840-f84e08dc496c-kube-api-access-58mfh\") pod \"213b86b6-9d1a-4e4a-8840-f84e08dc496c\" (UID: \"213b86b6-9d1a-4e4a-8840-f84e08dc496c\") "
Jul 09 16:59:13.663804 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:13.663780    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/213b86b6-9d1a-4e4a-8840-f84e08dc496c-kube-api-access-58mfh" (OuterVolumeSpecName: "kube-api-access-58mfh") pod "213b86b6-9d1a-4e4a-8840-f84e08dc496c" (UID: "213b86b6-9d1a-4e4a-8840-f84e08dc496c"). InnerVolumeSpecName "kube-api-access-58mfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:59:13.763251 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:13.763219    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58mfh\" (UniqueName: \"kubernetes.io/projected/213b86b6-9d1a-4e4a-8840-f84e08dc496c-kube-api-access-58mfh\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:59:14.367025 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.366990    2563 generic.go:356] "Generic (PLEG): container finished" podID="213b86b6-9d1a-4e4a-8840-f84e08dc496c" containerID="81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145" exitCode=0
Jul 09 16:59:14.367025 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.367028    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-l5fjt"
Jul 09 16:59:14.367470 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.367075    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5fjt" event={"ID":"213b86b6-9d1a-4e4a-8840-f84e08dc496c","Type":"ContainerDied","Data":"81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145"}
Jul 09 16:59:14.367470 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.367116    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-l5fjt" event={"ID":"213b86b6-9d1a-4e4a-8840-f84e08dc496c","Type":"ContainerDied","Data":"784071bfb9112b24c2e7d4f47f5c655494a2de0bf55330c489ec7fa00ce03033"}
Jul 09 16:59:14.367470 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.367133    2563 scope.go:117] "RemoveContainer" containerID="81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145"
Jul 09 16:59:14.375471 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.375451    2563 scope.go:117] "RemoveContainer" containerID="81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145"
Jul 09 16:59:14.375737 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:59:14.375718    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145\": container with ID starting with 81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145 not found: ID does not exist" containerID="81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145"
Jul 09 16:59:14.375789 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.375746    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145"} err="failed to get container status \"81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145\": rpc error: code = NotFound desc = could not find container \"81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145\": container with ID starting with 81adb3b31aa62486af60ba753b03ca32a86ed08073cd105197d52671703a5145 not found: ID does not exist"
Jul 09 16:59:14.387146 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.387122    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5fjt"]
Jul 09 16:59:14.390824 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:14.390803    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-l5fjt"]
Jul 09 16:59:15.312242 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:15.312206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:15.352426 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:15.352396    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="213b86b6-9d1a-4e4a-8840-f84e08dc496c" path="/var/lib/kubelet/pods/213b86b6-9d1a-4e4a-8840-f84e08dc496c/volumes"
Jul 09 16:59:15.416584 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:15.416551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:22.457760 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:22.457722    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:22.560879 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:22.560854    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:23.011388 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:23.011362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:35.881331 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:35.881288    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:36.103147 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.103110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:36.106423 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.106399    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jrnd2"]
Jul 09 16:59:36.106727 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.106715    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="213b86b6-9d1a-4e4a-8840-f84e08dc496c" containerName="authorino"
Jul 09 16:59:36.106772 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.106729    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="213b86b6-9d1a-4e4a-8840-f84e08dc496c" containerName="authorino"
Jul 09 16:59:36.106806 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.106776    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="213b86b6-9d1a-4e4a-8840-f84e08dc496c" containerName="authorino"
Jul 09 16:59:36.109422 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.109407    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jrnd2"
Jul 09 16:59:36.116708 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.116685    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jrnd2"]
Jul 09 16:59:36.236395 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.236303    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5krk\" (UniqueName: \"kubernetes.io/projected/919456fd-efd0-45b6-ba39-97ae39b58214-kube-api-access-l5krk\") pod \"authorino-5d4b67c446-jrnd2\" (UID: \"919456fd-efd0-45b6-ba39-97ae39b58214\") " pod="kuadrant-system/authorino-5d4b67c446-jrnd2"
Jul 09 16:59:36.288371 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.288337    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jrnd2"]
Jul 09 16:59:36.288581 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:59:36.288560    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-l5krk], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-5d4b67c446-jrnd2" podUID="919456fd-efd0-45b6-ba39-97ae39b58214"
Jul 09 16:59:36.314931 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.314898    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-76877994-8n5zv"]
Jul 09 16:59:36.317955 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.317939    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.320641 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.320620    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 09 16:59:36.328513 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.328493    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-76877994-8n5zv"]
Jul 09 16:59:36.337109 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.337073    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5krk\" (UniqueName: \"kubernetes.io/projected/919456fd-efd0-45b6-ba39-97ae39b58214-kube-api-access-l5krk\") pod \"authorino-5d4b67c446-jrnd2\" (UID: \"919456fd-efd0-45b6-ba39-97ae39b58214\") " pod="kuadrant-system/authorino-5d4b67c446-jrnd2"
Jul 09 16:59:36.345511 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.345484    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5krk\" (UniqueName: \"kubernetes.io/projected/919456fd-efd0-45b6-ba39-97ae39b58214-kube-api-access-l5krk\") pod \"authorino-5d4b67c446-jrnd2\" (UID: \"919456fd-efd0-45b6-ba39-97ae39b58214\") " pod="kuadrant-system/authorino-5d4b67c446-jrnd2"
Jul 09 16:59:36.352764 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.352740    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-8n5zv"]
Jul 09 16:59:36.352982 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:59:36.352961    2563 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-4csj8 tls-cert], unattached volumes=[], failed to process volumes=[kube-api-access-4csj8 tls-cert]: context canceled" pod="kuadrant-system/authorino-76877994-8n5zv" podUID="29c458fe-a330-43fc-8354-ba00572f7fc3"
Jul 09 16:59:36.381960 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.381927    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d565b5757-q6kpm"]
Jul 09 16:59:36.385081 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.385065    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.394370 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.394347    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d565b5757-q6kpm"]
Jul 09 16:59:36.438328 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.438295    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c458fe-a330-43fc-8354-ba00572f7fc3-tls-cert\") pod \"authorino-76877994-8n5zv\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") " pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.438498 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.438337    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f0153b9-5767-44bd-ad24-d2f69ed69781-tls-cert\") pod \"authorino-5d565b5757-q6kpm\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") " pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.438498 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.438380    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb692\" (UniqueName: \"kubernetes.io/projected/5f0153b9-5767-44bd-ad24-d2f69ed69781-kube-api-access-cb692\") pod \"authorino-5d565b5757-q6kpm\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") " pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.438498 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.438429    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4csj8\" (UniqueName: \"kubernetes.io/projected/29c458fe-a330-43fc-8354-ba00572f7fc3-kube-api-access-4csj8\") pod \"authorino-76877994-8n5zv\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") " pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.448549 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.448525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jrnd2"
Jul 09 16:59:36.448713 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.448525    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.455454 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.455433    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jrnd2"
Jul 09 16:59:36.458106 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.458089    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.539472 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.539377    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l5krk\" (UniqueName: \"kubernetes.io/projected/919456fd-efd0-45b6-ba39-97ae39b58214-kube-api-access-l5krk\") pod \"919456fd-efd0-45b6-ba39-97ae39b58214\" (UID: \"919456fd-efd0-45b6-ba39-97ae39b58214\") "
Jul 09 16:59:36.539637 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.539562    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c458fe-a330-43fc-8354-ba00572f7fc3-tls-cert\") pod \"authorino-76877994-8n5zv\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") " pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.539637 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.539605    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f0153b9-5767-44bd-ad24-d2f69ed69781-tls-cert\") pod \"authorino-5d565b5757-q6kpm\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") " pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.539792 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.539678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb692\" (UniqueName: \"kubernetes.io/projected/5f0153b9-5767-44bd-ad24-d2f69ed69781-kube-api-access-cb692\") pod \"authorino-5d565b5757-q6kpm\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") " pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.539792 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.539718    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4csj8\" (UniqueName: \"kubernetes.io/projected/29c458fe-a330-43fc-8354-ba00572f7fc3-kube-api-access-4csj8\") pod \"authorino-76877994-8n5zv\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") " pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.541611 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.541583    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/919456fd-efd0-45b6-ba39-97ae39b58214-kube-api-access-l5krk" (OuterVolumeSpecName: "kube-api-access-l5krk") pod "919456fd-efd0-45b6-ba39-97ae39b58214" (UID: "919456fd-efd0-45b6-ba39-97ae39b58214"). InnerVolumeSpecName "kube-api-access-l5krk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:59:36.541962 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.541943    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c458fe-a330-43fc-8354-ba00572f7fc3-tls-cert\") pod \"authorino-76877994-8n5zv\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") " pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.542076 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.542059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f0153b9-5767-44bd-ad24-d2f69ed69781-tls-cert\") pod \"authorino-5d565b5757-q6kpm\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") " pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.547160 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.547134    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4csj8\" (UniqueName: \"kubernetes.io/projected/29c458fe-a330-43fc-8354-ba00572f7fc3-kube-api-access-4csj8\") pod \"authorino-76877994-8n5zv\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") " pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:36.547275 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.547257    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb692\" (UniqueName: \"kubernetes.io/projected/5f0153b9-5767-44bd-ad24-d2f69ed69781-kube-api-access-cb692\") pod \"authorino-5d565b5757-q6kpm\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") " pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.640776 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.640741    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c458fe-a330-43fc-8354-ba00572f7fc3-tls-cert\") pod \"29c458fe-a330-43fc-8354-ba00572f7fc3\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") "
Jul 09 16:59:36.640913 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.640820    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4csj8\" (UniqueName: \"kubernetes.io/projected/29c458fe-a330-43fc-8354-ba00572f7fc3-kube-api-access-4csj8\") pod \"29c458fe-a330-43fc-8354-ba00572f7fc3\" (UID: \"29c458fe-a330-43fc-8354-ba00572f7fc3\") "
Jul 09 16:59:36.641055 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.641039    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5krk\" (UniqueName: \"kubernetes.io/projected/919456fd-efd0-45b6-ba39-97ae39b58214-kube-api-access-l5krk\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:59:36.642851 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.642821    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29c458fe-a330-43fc-8354-ba00572f7fc3-kube-api-access-4csj8" (OuterVolumeSpecName: "kube-api-access-4csj8") pod "29c458fe-a330-43fc-8354-ba00572f7fc3" (UID: "29c458fe-a330-43fc-8354-ba00572f7fc3"). InnerVolumeSpecName "kube-api-access-4csj8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:59:36.642851 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.642822    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29c458fe-a330-43fc-8354-ba00572f7fc3-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "29c458fe-a330-43fc-8354-ba00572f7fc3" (UID: "29c458fe-a330-43fc-8354-ba00572f7fc3"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 16:59:36.694197 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.694154    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 16:59:36.742034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.741988    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/29c458fe-a330-43fc-8354-ba00572f7fc3-tls-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:59:36.742034 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.742030    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4csj8\" (UniqueName: \"kubernetes.io/projected/29c458fe-a330-43fc-8354-ba00572f7fc3-kube-api-access-4csj8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:59:36.815360 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:36.815255    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d565b5757-q6kpm"]
Jul 09 16:59:36.817474 ip-10-0-139-29 kubenswrapper[2563]: W0709 16:59:36.817449    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5f0153b9_5767_44bd_ad24_d2f69ed69781.slice/crio-b1e886c1c0388e3e74a69726549be0a2ecdb38457ab5685302134a2cf110d6f6 WatchSource:0}: Error finding container b1e886c1c0388e3e74a69726549be0a2ecdb38457ab5685302134a2cf110d6f6: Status 404 returned error can't find the container with id b1e886c1c0388e3e74a69726549be0a2ecdb38457ab5685302134a2cf110d6f6
Jul 09 16:59:37.453561 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.453468    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d565b5757-q6kpm" event={"ID":"5f0153b9-5767-44bd-ad24-d2f69ed69781","Type":"ContainerStarted","Data":"6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7"}
Jul 09 16:59:37.453561 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.453512    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-jrnd2"
Jul 09 16:59:37.454001 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.453515    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-76877994-8n5zv"
Jul 09 16:59:37.454001 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.453518    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d565b5757-q6kpm" event={"ID":"5f0153b9-5767-44bd-ad24-d2f69ed69781","Type":"ContainerStarted","Data":"b1e886c1c0388e3e74a69726549be0a2ecdb38457ab5685302134a2cf110d6f6"}
Jul 09 16:59:37.473418 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.473373    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d565b5757-q6kpm" podStartSLOduration=1.143765719 podStartE2EDuration="1.473359964s" totalImagesPullingTime="329.594245ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 16:59:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 16:59:37.471431033 +0000 UTC m=+524.551371996" watchObservedRunningTime="2026-07-09 16:59:37.473359964 +0000 UTC m=+524.553300932"
Jul 09 16:59:37.503173 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.503146    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jrnd2"]
Jul 09 16:59:37.506151 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.506128    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vsxtg"]
Jul 09 16:59:37.506352 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.506326    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-vsxtg" podUID="1b8b856f-0daf-47b1-aa53-c0a87cfffbb0" containerName="authorino" containerID="cri-o://67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996" gracePeriod=30
Jul 09 16:59:37.509190 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.509163    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-jrnd2"]
Jul 09 16:59:37.535283 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.535259    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-76877994-8n5zv"]
Jul 09 16:59:37.538975 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.538953    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-76877994-8n5zv"]
Jul 09 16:59:37.744641 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.744619    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vsxtg"
Jul 09 16:59:37.850890 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.850851    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qs67p\" (UniqueName: \"kubernetes.io/projected/1b8b856f-0daf-47b1-aa53-c0a87cfffbb0-kube-api-access-qs67p\") pod \"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0\" (UID: \"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0\") "
Jul 09 16:59:37.852903 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.852876    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b8b856f-0daf-47b1-aa53-c0a87cfffbb0-kube-api-access-qs67p" (OuterVolumeSpecName: "kube-api-access-qs67p") pod "1b8b856f-0daf-47b1-aa53-c0a87cfffbb0" (UID: "1b8b856f-0daf-47b1-aa53-c0a87cfffbb0"). InnerVolumeSpecName "kube-api-access-qs67p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 16:59:37.951847 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:37.951802    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qs67p\" (UniqueName: \"kubernetes.io/projected/1b8b856f-0daf-47b1-aa53-c0a87cfffbb0-kube-api-access-qs67p\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 16:59:38.458229 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.458193    2563 generic.go:356] "Generic (PLEG): container finished" podID="1b8b856f-0daf-47b1-aa53-c0a87cfffbb0" containerID="67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996" exitCode=0
Jul 09 16:59:38.458670 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.458244    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-vsxtg"
Jul 09 16:59:38.458670 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.458277    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vsxtg" event={"ID":"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0","Type":"ContainerDied","Data":"67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996"}
Jul 09 16:59:38.458670 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.458322    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-vsxtg" event={"ID":"1b8b856f-0daf-47b1-aa53-c0a87cfffbb0","Type":"ContainerDied","Data":"48c6d3de89b8a885ca7d8c33519227c4c22d471932e20f879104d039535a349b"}
Jul 09 16:59:38.458670 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.458339    2563 scope.go:117] "RemoveContainer" containerID="67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996"
Jul 09 16:59:38.466250 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.466237    2563 scope.go:117] "RemoveContainer" containerID="67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996"
Jul 09 16:59:38.466456 ip-10-0-139-29 kubenswrapper[2563]: E0709 16:59:38.466440    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996\": container with ID starting with 67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996 not found: ID does not exist" containerID="67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996"
Jul 09 16:59:38.466500 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.466466    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996"} err="failed to get container status \"67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996\": rpc error: code = NotFound desc = could not find container \"67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996\": container with ID starting with 67d808885c0da59c8e3a96a06ddb74a429ae4515d52b7d5e1e74cf766ff4f996 not found: ID does not exist"
Jul 09 16:59:38.477918 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.477899    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vsxtg"]
Jul 09 16:59:38.483790 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:38.483772    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-vsxtg"]
Jul 09 16:59:39.352337 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:39.352308    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b8b856f-0daf-47b1-aa53-c0a87cfffbb0" path="/var/lib/kubelet/pods/1b8b856f-0daf-47b1-aa53-c0a87cfffbb0/volumes"
Jul 09 16:59:39.352667 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:39.352638    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="29c458fe-a330-43fc-8354-ba00572f7fc3" path="/var/lib/kubelet/pods/29c458fe-a330-43fc-8354-ba00572f7fc3/volumes"
Jul 09 16:59:39.352845 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:39.352834    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="919456fd-efd0-45b6-ba39-97ae39b58214" path="/var/lib/kubelet/pods/919456fd-efd0-45b6-ba39-97ae39b58214/volumes"
Jul 09 16:59:41.653919 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:41.653879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:41.721454 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:41.721424    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:42.594369 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:42.594336    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:45.343718 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:45.343622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:45.436400 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:45.436362    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:50.178140 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:50.178106    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 16:59:50.279123 ip-10-0-139-29 kubenswrapper[2563]: I0709 16:59:50.279087    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:11.204609 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:11.204569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:16.736607 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:16.736566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:17.265905 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:17.265873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:17.401138 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:17.401108    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:17.749172 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:17.749132    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:17.839193 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:17.839165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:29.879030 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:29.878997    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:29.979081 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:29.979049    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:40.992095 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.992039    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"]
Jul 09 17:00:40.992592 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.992323    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b8b856f-0daf-47b1-aa53-c0a87cfffbb0" containerName="authorino"
Jul 09 17:00:40.992592 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.992335    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b8b856f-0daf-47b1-aa53-c0a87cfffbb0" containerName="authorino"
Jul 09 17:00:40.992592 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.992387    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b8b856f-0daf-47b1-aa53-c0a87cfffbb0" containerName="authorino"
Jul 09 17:00:40.995349 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.995326    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:40.997857 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.997835    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 17:00:40.997964 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.997889    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Jul 09 17:00:40.998911 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.998894    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 17:00:40.998988 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:40.998975    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-ccqbg\""
Jul 09 17:00:41.004878 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.004854    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"]
Jul 09 17:00:41.019152 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.019123    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:41.126405 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.126365    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.126590 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.126411    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlgx5\" (UniqueName: \"kubernetes.io/projected/5d594c3e-1304-431c-8b24-84ba41017b74-kube-api-access-dlgx5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.126590 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.126491    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.126590 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.126529    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.126590 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.126551    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.126590 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.126585    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d594c3e-1304-431c-8b24-84ba41017b74-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227488 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227450    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d594c3e-1304-431c-8b24-84ba41017b74-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227691 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227511    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227691 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227530    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlgx5\" (UniqueName: \"kubernetes.io/projected/5d594c3e-1304-431c-8b24-84ba41017b74-kube-api-access-dlgx5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227691 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227555    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227691 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227575    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227691 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227595    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227933 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227887    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.227996 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.227933    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-home\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.228063 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.228046    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-model-cache\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.229860 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.229840    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5d594c3e-1304-431c-8b24-84ba41017b74-dshm\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.230035 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.230016    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5d594c3e-1304-431c-8b24-84ba41017b74-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.234931 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.234910    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlgx5\" (UniqueName: \"kubernetes.io/projected/5d594c3e-1304-431c-8b24-84ba41017b74-kube-api-access-dlgx5\") pod \"e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7\" (UID: \"5d594c3e-1304-431c-8b24-84ba41017b74\") " pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.305616 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.305520    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:41.433401 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.433367    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"]
Jul 09 17:00:41.436714 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:00:41.436682    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d594c3e_1304_431c_8b24_84ba41017b74.slice/crio-da1807a428abcc605fa23df2a181909aa08993d8e329ea32b95261970d318fb5 WatchSource:0}: Error finding container da1807a428abcc605fa23df2a181909aa08993d8e329ea32b95261970d318fb5: Status 404 returned error can't find the container with id da1807a428abcc605fa23df2a181909aa08993d8e329ea32b95261970d318fb5
Jul 09 17:00:41.694354 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:41.694314    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7" event={"ID":"5d594c3e-1304-431c-8b24-84ba41017b74","Type":"ContainerStarted","Data":"da1807a428abcc605fa23df2a181909aa08993d8e329ea32b95261970d318fb5"}
Jul 09 17:00:42.700497 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:42.700466    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7" event={"ID":"5d594c3e-1304-431c-8b24-84ba41017b74","Type":"ContainerStarted","Data":"c14401f8f827bbe2aecfe1a3da284fb7f8e71fa54ab66019eed7c22fc02e6010"}
Jul 09 17:00:42.729297 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:42.729245    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7" podStartSLOduration=1.6346562599999999 podStartE2EDuration="2.729228382s" totalImagesPullingTime="1.094572122s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:00:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:00:42.728524202 +0000 UTC m=+589.808465171" watchObservedRunningTime="2026-07-09 17:00:42.729228382 +0000 UTC m=+589.809169350"
Jul 09 17:00:50.071984 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:50.071950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:00:51.307706 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:51.307673    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:51.309407 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:51.309376    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:51.325099 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:51.325070    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:00:51.746346 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:00:51.746307    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7"
Jul 09 17:01:03.219093 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:03.219056    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:01:33.953710 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:33.953674    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:01:42.793704 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.793637    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"]
Jul 09 17:01:42.796935 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.796913    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.799518 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.799497    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 09 17:01:42.805387 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.805364    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"]
Jul 09 17:01:42.836579 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.836543    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.836579 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.836581    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgghk\" (UniqueName: \"kubernetes.io/projected/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-kube-api-access-wgghk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.836800 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.836602    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.836800 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.836678    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.836800 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.836706    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.836800 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.836723    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938061 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938026    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgghk\" (UniqueName: \"kubernetes.io/projected/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-kube-api-access-wgghk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938061 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938066    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938331 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938103    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938331 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938135    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938331 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938154    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938331 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938201    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938536 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938451    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938591 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938539    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.938591 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.938559    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.940501 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.940474    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.940725 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.940703    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:42.946211 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:42.946182    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgghk\" (UniqueName: \"kubernetes.io/projected/e6958f86-fd5e-4c6a-9ca5-37782a3a303c-kube-api-access-wgghk\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-vcwpt\" (UID: \"e6958f86-fd5e-4c6a-9ca5-37782a3a303c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:43.107403 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:43.107375    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:43.239924 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:43.239764    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"]
Jul 09 17:01:43.242179 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:01:43.242146    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6958f86_fd5e_4c6a_9ca5_37782a3a303c.slice/crio-1dc3fd554773d1a4859acc15122b9c0f6474205a8b71b624f5cf6f1e4e88b4d8 WatchSource:0}: Error finding container 1dc3fd554773d1a4859acc15122b9c0f6474205a8b71b624f5cf6f1e4e88b4d8: Status 404 returned error can't find the container with id 1dc3fd554773d1a4859acc15122b9c0f6474205a8b71b624f5cf6f1e4e88b4d8
Jul 09 17:01:43.243521 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:43.243500    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:01:43.933633 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:43.933595    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt" event={"ID":"e6958f86-fd5e-4c6a-9ca5-37782a3a303c","Type":"ContainerStarted","Data":"23d7ab8ea4808a6ca038264de3a1aea8bce55fabdf42d3a9976087a21aff08f9"}
Jul 09 17:01:43.934022 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:43.933679    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt" event={"ID":"e6958f86-fd5e-4c6a-9ca5-37782a3a303c","Type":"ContainerStarted","Data":"1dc3fd554773d1a4859acc15122b9c0f6474205a8b71b624f5cf6f1e4e88b4d8"}
Jul 09 17:01:43.951804 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:43.951750    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt" podStartSLOduration=1.668901097 podStartE2EDuration="1.951730197s" totalImagesPullingTime="282.8291ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:01:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:01:43.950178069 +0000 UTC m=+651.030119069" watchObservedRunningTime="2026-07-09 17:01:43.951730197 +0000 UTC m=+651.031671168"
Jul 09 17:01:44.318465 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:44.318379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:01:50.491675 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.491627    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"]
Jul 09 17:01:50.494837 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.494819    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.497323 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.497299    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Jul 09 17:01:50.505166 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.505143    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"]
Jul 09 17:01:50.601642 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.601607    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c94d5d-7b3a-4046-91d7-e75a8d046474-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.601816 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.601677    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.601816 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.601703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.601816 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.601736    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.601816 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.601787    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.601816 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.601805    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2f25k\" (UniqueName: \"kubernetes.io/projected/a1c94d5d-7b3a-4046-91d7-e75a8d046474-kube-api-access-2f25k\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.679283 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.679237    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:01:50.702469 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702436    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c94d5d-7b3a-4046-91d7-e75a8d046474-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.702587 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.702587 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702509    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.702587 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702544    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.702587 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702571    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.702797 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702590    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2f25k\" (UniqueName: \"kubernetes.io/projected/a1c94d5d-7b3a-4046-91d7-e75a8d046474-kube-api-access-2f25k\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.702989 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702962    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-home\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.703048 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.702985    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-model-cache\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.703086 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.703056    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.704907 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.704874    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a1c94d5d-7b3a-4046-91d7-e75a8d046474-dshm\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.705128 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.705111    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a1c94d5d-7b3a-4046-91d7-e75a8d046474-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.710795 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.710774    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2f25k\" (UniqueName: \"kubernetes.io/projected/a1c94d5d-7b3a-4046-91d7-e75a8d046474-kube-api-access-2f25k\") pod \"premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm\" (UID: \"a1c94d5d-7b3a-4046-91d7-e75a8d046474\") " pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.804059 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.803965    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:01:50.936037 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.936012    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"]
Jul 09 17:01:50.938748 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:01:50.938720    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda1c94d5d_7b3a_4046_91d7_e75a8d046474.slice/crio-5142029d24adbdb38ebac7149673309b5052e1ab4d37204531ea0e9174b028f6 WatchSource:0}: Error finding container 5142029d24adbdb38ebac7149673309b5052e1ab4d37204531ea0e9174b028f6: Status 404 returned error can't find the container with id 5142029d24adbdb38ebac7149673309b5052e1ab4d37204531ea0e9174b028f6
Jul 09 17:01:50.965695 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:50.965648    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm" event={"ID":"a1c94d5d-7b3a-4046-91d7-e75a8d046474","Type":"ContainerStarted","Data":"5142029d24adbdb38ebac7149673309b5052e1ab4d37204531ea0e9174b028f6"}
Jul 09 17:01:51.971591 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:51.971552    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm" event={"ID":"a1c94d5d-7b3a-4046-91d7-e75a8d046474","Type":"ContainerStarted","Data":"3ac7ccca9f87dd6f9426c53398c14248efa34c15293e1a64f7e2ec0eb8b777b5"}
Jul 09 17:01:51.988544 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:51.988499    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm" podStartSLOduration=1.704517353 podStartE2EDuration="1.98848481s" totalImagesPullingTime="283.967457ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:01:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:01:51.98754788 +0000 UTC m=+659.067488873" watchObservedRunningTime="2026-07-09 17:01:51.98848481 +0000 UTC m=+659.068425778"
Jul 09 17:01:53.108501 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:53.108462    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:53.108971 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:53.108555    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:53.121204 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:53.121180    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:01:53.996932 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:01:53.996897    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-vcwpt"
Jul 09 17:02:00.804461 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:00.804413    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:02:00.804461 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:00.804461    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:02:00.816925 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:00.816895    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:02:01.024358 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:01.024317    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm"
Jul 09 17:02:13.613553 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:13.613518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:13.713275 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:13.713242    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:19.612895 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:19.612859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:36.827563 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.827527    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5dfb7d9596-dwlht"]
Jul 09 17:02:36.830767 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.830749    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:36.839819 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.839794    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dfb7d9596-dwlht"]
Jul 09 17:02:36.877950 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.877905    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb800722-ecc1-421e-8627-d7d086f2c9af-tls-cert\") pod \"authorino-5dfb7d9596-dwlht\" (UID: \"fb800722-ecc1-421e-8627-d7d086f2c9af\") " pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:36.878184 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.877958    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fb800722-ecc1-421e-8627-d7d086f2c9af-kube-api-access-nxbdb\") pod \"authorino-5dfb7d9596-dwlht\" (UID: \"fb800722-ecc1-421e-8627-d7d086f2c9af\") " pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:36.978799 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.978762    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb800722-ecc1-421e-8627-d7d086f2c9af-tls-cert\") pod \"authorino-5dfb7d9596-dwlht\" (UID: \"fb800722-ecc1-421e-8627-d7d086f2c9af\") " pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:36.978962 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.978812    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fb800722-ecc1-421e-8627-d7d086f2c9af-kube-api-access-nxbdb\") pod \"authorino-5dfb7d9596-dwlht\" (UID: \"fb800722-ecc1-421e-8627-d7d086f2c9af\") " pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:36.981213 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.981188    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb800722-ecc1-421e-8627-d7d086f2c9af-tls-cert\") pod \"authorino-5dfb7d9596-dwlht\" (UID: \"fb800722-ecc1-421e-8627-d7d086f2c9af\") " pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:36.986755 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:36.986730    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxbdb\" (UniqueName: \"kubernetes.io/projected/fb800722-ecc1-421e-8627-d7d086f2c9af-kube-api-access-nxbdb\") pod \"authorino-5dfb7d9596-dwlht\" (UID: \"fb800722-ecc1-421e-8627-d7d086f2c9af\") " pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:37.140112 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:37.140080    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5dfb7d9596-dwlht"
Jul 09 17:02:37.194009 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:37.193978    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:37.274430 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:37.274391    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5dfb7d9596-dwlht"]
Jul 09 17:02:37.277320 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:02:37.277287    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb800722_ecc1_421e_8627_d7d086f2c9af.slice/crio-947cfa5e4f145df5c4ef2fab1a42c153615e38fdc4f227ede4019e7fd2df5116 WatchSource:0}: Error finding container 947cfa5e4f145df5c4ef2fab1a42c153615e38fdc4f227ede4019e7fd2df5116: Status 404 returned error can't find the container with id 947cfa5e4f145df5c4ef2fab1a42c153615e38fdc4f227ede4019e7fd2df5116
Jul 09 17:02:37.277511 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:37.277488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:38.165858 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.165597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfb7d9596-dwlht" event={"ID":"fb800722-ecc1-421e-8627-d7d086f2c9af","Type":"ContainerStarted","Data":"e4e8c1193c624ca3d9f76e824234abc5efc3e3abf9f113b8da0bfa02aabed2af"}
Jul 09 17:02:38.165858 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.165646    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5dfb7d9596-dwlht" event={"ID":"fb800722-ecc1-421e-8627-d7d086f2c9af","Type":"ContainerStarted","Data":"947cfa5e4f145df5c4ef2fab1a42c153615e38fdc4f227ede4019e7fd2df5116"}
Jul 09 17:02:38.181894 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.181829    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5dfb7d9596-dwlht" podStartSLOduration=1.693412905 podStartE2EDuration="2.181804803s" totalImagesPullingTime="488.391898ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:02:38.181310281 +0000 UTC m=+705.261251251" watchObservedRunningTime="2026-07-09 17:02:38.181804803 +0000 UTC m=+705.261745773"
Jul 09 17:02:38.217571 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.217531    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d565b5757-q6kpm"]
Jul 09 17:02:38.217803 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.217777    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d565b5757-q6kpm" podUID="5f0153b9-5767-44bd-ad24-d2f69ed69781" containerName="authorino" containerID="cri-o://6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7" gracePeriod=30
Jul 09 17:02:38.320271 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.320244    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:38.421315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.421100    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:38.479286 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.479247    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 17:02:38.594368 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.594339    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f0153b9-5767-44bd-ad24-d2f69ed69781-tls-cert\") pod \"5f0153b9-5767-44bd-ad24-d2f69ed69781\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") "
Jul 09 17:02:38.594542 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.594395    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cb692\" (UniqueName: \"kubernetes.io/projected/5f0153b9-5767-44bd-ad24-d2f69ed69781-kube-api-access-cb692\") pod \"5f0153b9-5767-44bd-ad24-d2f69ed69781\" (UID: \"5f0153b9-5767-44bd-ad24-d2f69ed69781\") "
Jul 09 17:02:38.596591 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.596561    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f0153b9-5767-44bd-ad24-d2f69ed69781-kube-api-access-cb692" (OuterVolumeSpecName: "kube-api-access-cb692") pod "5f0153b9-5767-44bd-ad24-d2f69ed69781" (UID: "5f0153b9-5767-44bd-ad24-d2f69ed69781"). InnerVolumeSpecName "kube-api-access-cb692". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:02:38.604372 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.604341    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f0153b9-5767-44bd-ad24-d2f69ed69781-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "5f0153b9-5767-44bd-ad24-d2f69ed69781" (UID: "5f0153b9-5767-44bd-ad24-d2f69ed69781"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:02:38.696025 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.695946    2563 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5f0153b9-5767-44bd-ad24-d2f69ed69781-tls-cert\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:02:38.696025 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:38.695973    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cb692\" (UniqueName: \"kubernetes.io/projected/5f0153b9-5767-44bd-ad24-d2f69ed69781-kube-api-access-cb692\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:02:39.171391 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.171353    2563 generic.go:356] "Generic (PLEG): container finished" podID="5f0153b9-5767-44bd-ad24-d2f69ed69781" containerID="6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7" exitCode=0
Jul 09 17:02:39.171829 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.171403    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d565b5757-q6kpm"
Jul 09 17:02:39.171829 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.171444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d565b5757-q6kpm" event={"ID":"5f0153b9-5767-44bd-ad24-d2f69ed69781","Type":"ContainerDied","Data":"6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7"}
Jul 09 17:02:39.171829 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.171488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d565b5757-q6kpm" event={"ID":"5f0153b9-5767-44bd-ad24-d2f69ed69781","Type":"ContainerDied","Data":"b1e886c1c0388e3e74a69726549be0a2ecdb38457ab5685302134a2cf110d6f6"}
Jul 09 17:02:39.171829 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.171505    2563 scope.go:117] "RemoveContainer" containerID="6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7"
Jul 09 17:02:39.180519 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.180499    2563 scope.go:117] "RemoveContainer" containerID="6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7"
Jul 09 17:02:39.180836 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:02:39.180815    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7\": container with ID starting with 6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7 not found: ID does not exist" containerID="6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7"
Jul 09 17:02:39.180916 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.180844    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7"} err="failed to get container status \"6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7\": rpc error: code = NotFound desc = could not find container \"6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7\": container with ID starting with 6bbf93d6a656d1551c858b012a31e6e02a8a0fcaecc10f6ea48a024069ea0ae7 not found: ID does not exist"
Jul 09 17:02:39.201420 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.201375    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d565b5757-q6kpm"]
Jul 09 17:02:39.201834 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.201808    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d565b5757-q6kpm"]
Jul 09 17:02:39.354312 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:39.354214    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5f0153b9-5767-44bd-ad24-d2f69ed69781" path="/var/lib/kubelet/pods/5f0153b9-5767-44bd-ad24-d2f69ed69781/volumes"
Jul 09 17:02:41.948823 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:41.948777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:02:42.049205 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:02:42.049168    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:03:48.280768 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:03:48.280730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:03:49.649375 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:03:49.649338    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:31.231314 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:31.231220    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:31.324849 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:31.324814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:31.459017 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:31.458984    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:37.232282 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:37.232246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:37.340865 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:37.340834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:37.439342 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:37.439306    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:42.867676 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:42.867629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:42.930733 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:42.930703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:50.240056 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:50.240023    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:04:57.840090 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:04:57.840057    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:05:05.476979 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:05:05.476944    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:07:10.302990 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:07:10.302959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:09:06.635426 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:09:06.635393    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:09:06.707533 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:09:06.707502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:11:05.637165 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:11:05.637131    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:13:12.187842 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:13:12.187807    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:13:14.258195 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:13:14.258165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:15:32.918455 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:15:32.918422    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:15:41.271078 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:15:41.271043    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:15:48.550420 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:15:48.550390    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:15:54.082552 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:15:54.082518    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:15:56.354009 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:15:56.353977    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:15:56.456533 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:15:56.456506    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:15:56.857755 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:15:56.857704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:02.487290 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:02.487257    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:02.619393 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:02.619347    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:02.704044 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:02.704011    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:02.769878 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:02.769797    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:03.808536 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:03.808474    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:03.898290 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:03.898259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:03.988705 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:03.988668    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:04.453078 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:04.453051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:04.557017 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:04.556991    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:04.662137 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:04.662105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:06.459433 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:06.459408    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:08.898859 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:08.898826    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:09.096114 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:09.096083    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:09.184128 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:09.184047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:09.285780 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:09.285749    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:10.477550 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:10.477525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:14.792969 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:14.792933    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:14.895812 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:14.895784    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:15.023107 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:15.023081    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:15.123113 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:15.123082    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:16.200513 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:16.200480    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:16.294824 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:16.294791    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:16.386368 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:16.386329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:16.482794 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:16.482704    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:16.871804 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:16.871773    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:16.982706 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:16.982675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:18.534248 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:18.534221    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:21.190863 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:21.190831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:21.618119 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:21.618089    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:23.503246 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:23.503213    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:27.278532 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:27.278499    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:27.379166 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:27.379135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:28.571764 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:28.571734    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:33.021241 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:33.021207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:33.595181 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:33.595134    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:33.695613 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:33.695582    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:34.339372 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:34.339343    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:34.440034 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:34.440005    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:34.531923 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:34.531887    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:34.705200 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:34.705164    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:35.019713 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:35.019608    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:35.119001 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:35.118968    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:35.232581 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:35.232548    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:36.396259 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:36.396224    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:38.787489 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:38.787451    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:38.938167 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:38.938133    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:39.038448 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:39.038370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:39.217807 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:39.217775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:40.732470 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:40.732442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:44.837905 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:44.837870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:44.937878 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:44.937847    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:45.290852 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:45.290819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:45.388604 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:45.388566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:45.490323 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:45.490273    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:46.855460 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:46.855431    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:53.012940 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:53.012909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:53.170643 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:53.170601    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:16:53.273899 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:16:53.273796    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:19.225012 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:19.224973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:24.878474 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:24.878442    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:24.940783 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:24.940750    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:25.027588 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:25.027549    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:25.126388 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:25.126358    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:39.052008 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:39.051973    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:40.969894 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:40.969859    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:46.797434 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:46.797351    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:46.909501 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:46.909469    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:46.997811 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:46.997777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:57.501070 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:57.501032    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:17:58.270912 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:17:58.270881    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:04.125817 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:04.125777    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:05.814066 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:05.814033    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:05.912794 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:05.912758    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:07.031761 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:07.031675    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:13.100934 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:13.100895    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:13.260881 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:13.260852    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:13.742940 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:13.742888    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:19.671313 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:19.671282    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:22.873146 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:22.873115    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:23.506495 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:23.506457    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:23.589939 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:23.589909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:23.660543 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:23.660512    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:23.984933 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:23.984904    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:24.137227 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:24.137192    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:24.238139 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:24.238059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:27.964769 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:27.964737    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:28.600129 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:28.600097    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:28.696645 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:28.696611    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:29.011793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:29.011756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:34.760236 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:34.760207    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:34.860168 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:34.860138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:35.162275 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:35.162238    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:35.563685 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:35.563550    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:35.661149 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:35.661117    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:41.482084 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:41.482052    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:41.579206 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:41.579177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:41.753952 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:41.753875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:41.967061 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:41.967030    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:42.075580 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:42.075505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:42.173540 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:42.173510    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:42.283518 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:42.283488    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:48.191496 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:48.191460    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:48.253795 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:48.253766    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:48.344186 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:48.344151    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:48.411161 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:48.411130    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:48.524991 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:48.524899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:54.186284 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:54.186255    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:18:54.244232 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:18:54.244202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:00.880354 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:00.880319    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:02.120552 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:02.120522    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:02.222028 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:02.221999    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:03.717628 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:03.717575    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:03.816242 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:03.816210    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:04.044364 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:04.044289    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:04.218344 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:04.218316    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:04.546938 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:04.546909    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:04.653624 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:04.653593    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:08.212751 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:08.212703    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:11.039257 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:11.039219    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:11.139758 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:11.139720    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:22.088771 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.088694    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"]
Jul 09 17:19:22.089140 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.088992    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5f0153b9-5767-44bd-ad24-d2f69ed69781" containerName="authorino"
Jul 09 17:19:22.089140 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.089007    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5f0153b9-5767-44bd-ad24-d2f69ed69781" containerName="authorino"
Jul 09 17:19:22.089140 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.089087    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5f0153b9-5767-44bd-ad24-d2f69ed69781" containerName="authorino"
Jul 09 17:19:22.091955 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.091940    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.096382 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.096350    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b9b28345\"/\"e2e-lifecycle-model-b9b28345-kserve-self-signed-certs\""
Jul 09 17:19:22.096517 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.096394    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b9b28345\"/\"openshift-service-ca.crt\""
Jul 09 17:19:22.096517 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.096481    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-b9b28345\"/\"default-dockercfg-blzmg\""
Jul 09 17:19:22.096620 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.096548    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-b9b28345\"/\"kube-root-ca.crt\""
Jul 09 17:19:22.104982 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.104961    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"]
Jul 09 17:19:22.116074 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.116047    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:22.163595 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.163572    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f59fa-913b-41d5-869c-687d01b81f4c-tls-certs\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.163752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.163637    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcc4f\" (UniqueName: \"kubernetes.io/projected/0f3f59fa-913b-41d5-869c-687d01b81f4c-kube-api-access-fcc4f\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.163752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.163696    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-tmp-dir\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.163752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.163725    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-model-cache\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.163859 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.163752    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-home\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.163859 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.163775    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-dshm\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.264978 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.264944    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcc4f\" (UniqueName: \"kubernetes.io/projected/0f3f59fa-913b-41d5-869c-687d01b81f4c-kube-api-access-fcc4f\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265180 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.264995    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-tmp-dir\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265180 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.265017    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-model-cache\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265180 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.265038    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-home\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265180 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.265061    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-dshm\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265180 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.265080    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f59fa-913b-41d5-869c-687d01b81f4c-tls-certs\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265486 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.265460    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-model-cache\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265547 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.265503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-home\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.265547 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.265519    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-tmp-dir\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.267297 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.267278    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-dshm\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.267560 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.267543    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f59fa-913b-41d5-869c-687d01b81f4c-tls-certs\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.272991 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.272965    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcc4f\" (UniqueName: \"kubernetes.io/projected/0f3f59fa-913b-41d5-869c-687d01b81f4c-kube-api-access-fcc4f\") pod \"e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") " pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.404534 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.404504    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:22.531231 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:22.531208    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"]
Jul 09 17:19:22.533896 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:19:22.533861    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f3f59fa_913b_41d5_869c_687d01b81f4c.slice/crio-73bd809c22e1172f80eb0294ca09bd6e9379fb4d9ac72fb9e5085d444b7c047d WatchSource:0}: Error finding container 73bd809c22e1172f80eb0294ca09bd6e9379fb4d9ac72fb9e5085d444b7c047d: Status 404 returned error can't find the container with id 73bd809c22e1172f80eb0294ca09bd6e9379fb4d9ac72fb9e5085d444b7c047d
Jul 09 17:19:23.202520 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:23.202486    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s" event={"ID":"0f3f59fa-913b-41d5-869c-687d01b81f4c","Type":"ContainerStarted","Data":"3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5"}
Jul 09 17:19:23.202931 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:23.202536    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s" event={"ID":"0f3f59fa-913b-41d5-869c-687d01b81f4c","Type":"ContainerStarted","Data":"73bd809c22e1172f80eb0294ca09bd6e9379fb4d9ac72fb9e5085d444b7c047d"}
Jul 09 17:19:23.219596 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:23.219552    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s" podStartSLOduration=1.219536825 podStartE2EDuration="1.219536825s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:19:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:19:23.218278202 +0000 UTC m=+1710.298219173" watchObservedRunningTime="2026-07-09 17:19:23.219536825 +0000 UTC m=+1710.299477836"
Jul 09 17:19:25.395857 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:25.395820    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:25.496626 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:25.496594    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:32.404674 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:32.404601    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:32.405105 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:32.404698    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:32.417468 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:32.417442    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:33.252912 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:33.252878    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:19:45.688921 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:45.688890    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:46.024894 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:46.024819    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:51.763501 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:51.763471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:19:56.742931 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:56.742893    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"]
Jul 09 17:19:56.786263 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:19:56.786225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:11.762870 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:11.762828    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s" podUID="0f3f59fa-913b-41d5-869c-687d01b81f4c" containerName="main" containerID="cri-o://3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5" gracePeriod=45
Jul 09 17:20:12.003914 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.003883    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:20:12.065152 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065075    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-tmp-dir\") pod \"0f3f59fa-913b-41d5-869c-687d01b81f4c\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") "
Jul 09 17:20:12.065152 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065134    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcc4f\" (UniqueName: \"kubernetes.io/projected/0f3f59fa-913b-41d5-869c-687d01b81f4c-kube-api-access-fcc4f\") pod \"0f3f59fa-913b-41d5-869c-687d01b81f4c\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") "
Jul 09 17:20:12.065349 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065156    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-home\") pod \"0f3f59fa-913b-41d5-869c-687d01b81f4c\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") "
Jul 09 17:20:12.065349 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065177    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-model-cache\") pod \"0f3f59fa-913b-41d5-869c-687d01b81f4c\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") "
Jul 09 17:20:12.065349 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065218    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f59fa-913b-41d5-869c-687d01b81f4c-tls-certs\") pod \"0f3f59fa-913b-41d5-869c-687d01b81f4c\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") "
Jul 09 17:20:12.065349 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065243    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-dshm\") pod \"0f3f59fa-913b-41d5-869c-687d01b81f4c\" (UID: \"0f3f59fa-913b-41d5-869c-687d01b81f4c\") "
Jul 09 17:20:12.065549 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065361    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f3f59fa-913b-41d5-869c-687d01b81f4c" (UID: "0f3f59fa-913b-41d5-869c-687d01b81f4c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:20:12.065549 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065397    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-home" (OuterVolumeSpecName: "home") pod "0f3f59fa-913b-41d5-869c-687d01b81f4c" (UID: "0f3f59fa-913b-41d5-869c-687d01b81f4c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:20:12.065549 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065492    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f3f59fa-913b-41d5-869c-687d01b81f4c" (UID: "0f3f59fa-913b-41d5-869c-687d01b81f4c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:20:12.065708 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065550    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:20:12.065708 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.065573    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:20:12.067494 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.067471    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-dshm" (OuterVolumeSpecName: "dshm") pod "0f3f59fa-913b-41d5-869c-687d01b81f4c" (UID: "0f3f59fa-913b-41d5-869c-687d01b81f4c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:20:12.067494 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.067482    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f3f59fa-913b-41d5-869c-687d01b81f4c-kube-api-access-fcc4f" (OuterVolumeSpecName: "kube-api-access-fcc4f") pod "0f3f59fa-913b-41d5-869c-687d01b81f4c" (UID: "0f3f59fa-913b-41d5-869c-687d01b81f4c"). InnerVolumeSpecName "kube-api-access-fcc4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:20:12.067705 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.067578    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f3f59fa-913b-41d5-869c-687d01b81f4c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f3f59fa-913b-41d5-869c-687d01b81f4c" (UID: "0f3f59fa-913b-41d5-869c-687d01b81f4c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:20:12.166397 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.166367    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcc4f\" (UniqueName: \"kubernetes.io/projected/0f3f59fa-913b-41d5-869c-687d01b81f4c-kube-api-access-fcc4f\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:20:12.166397 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.166395    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:20:12.166397 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.166406    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f3f59fa-913b-41d5-869c-687d01b81f4c-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:20:12.166613 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.166414    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f3f59fa-913b-41d5-869c-687d01b81f4c-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:20:12.397681 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.397632    2563 generic.go:356] "Generic (PLEG): container finished" podID="0f3f59fa-913b-41d5-869c-687d01b81f4c" containerID="3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5" exitCode=0
Jul 09 17:20:12.397839 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.397692    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"
Jul 09 17:20:12.397839 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.397721    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s" event={"ID":"0f3f59fa-913b-41d5-869c-687d01b81f4c","Type":"ContainerDied","Data":"3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5"}
Jul 09 17:20:12.397839 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.397768    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s" event={"ID":"0f3f59fa-913b-41d5-869c-687d01b81f4c","Type":"ContainerDied","Data":"73bd809c22e1172f80eb0294ca09bd6e9379fb4d9ac72fb9e5085d444b7c047d"}
Jul 09 17:20:12.397839 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.397786    2563 scope.go:117] "RemoveContainer" containerID="3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5"
Jul 09 17:20:12.406642 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.406625    2563 scope.go:117] "RemoveContainer" containerID="3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5"
Jul 09 17:20:12.406918 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:20:12.406900    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5\": container with ID starting with 3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5 not found: ID does not exist" containerID="3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5"
Jul 09 17:20:12.406971 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.406926    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5"} err="failed to get container status \"3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5\": rpc error: code = NotFound desc = could not find container \"3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5\": container with ID starting with 3a0bace1b7c12af240794a1c6f5bf4d4baaedb2a3f97d058027dd921f958d0d5 not found: ID does not exist"
Jul 09 17:20:12.417726 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.417704    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"]
Jul 09 17:20:12.422836 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:12.422812    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-b9b28345/e2e-lifecycle-model-b9b28345-kserve-78574b9cf9-xrr9s"]
Jul 09 17:20:13.353502 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:13.353472    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f3f59fa-913b-41d5-869c-687d01b81f4c" path="/var/lib/kubelet/pods/0f3f59fa-913b-41d5-869c-687d01b81f4c/volumes"
Jul 09 17:20:33.142545 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:33.142505    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:33.718443 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:33.718409    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:39.196486 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:39.196450    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:39.296595 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:39.296557    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:39.476499 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:39.476416    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:39.712076 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:39.712046    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:40.826132 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:40.826104    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:40.935999 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:40.935966    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:41.696848 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:41.696817    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:41.798335 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:41.798301    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:41.895357 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:41.895326    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:43.239908 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:43.239874    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:49.107907 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:49.107822    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:49.181697 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:49.181648    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:49.792756 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:49.792719    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:55.544907 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:55.544867    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:55.602906 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:55.602872    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:55.681091 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:55.681045    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:55.778109 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:55.778069    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:20:59.983829 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:20:59.983789    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:00.083923 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:00.083875    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:06.244171 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:06.244138    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:06.584261 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:06.584173    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:27.517680 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:27.517637    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:33.183846 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:33.183812    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:33.282206 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:33.282177    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:33.552265 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:33.552182    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:33.839838 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:33.839756    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:33.941804 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:33.941775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:36.545812 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:36.545776    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:36.636600 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:36.636569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:36.745753 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:36.745710    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:36.916328 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:36.916259    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:37.310273 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:37.310186    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:37.412626 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:37.412596    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:41.047574 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:41.047543    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:41.415954 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:41.415930    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:41.695310 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:41.695218    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:41.975685 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:41.975569    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:44.550142 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:44.550109    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:44.655428 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:44.655398    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:44.784482 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:44.784447    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:44.892214 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:44.892185    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:45.218716 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:45.218613    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:45.319233 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:45.319203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:49.044360 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:49.044329    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:55.278216 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.278185    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"]
Jul 09 17:21:55.278592 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.278447    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f3f59fa-913b-41d5-869c-687d01b81f4c" containerName="main"
Jul 09 17:21:55.278592 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.278457    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f3f59fa-913b-41d5-869c-687d01b81f4c" containerName="main"
Jul 09 17:21:55.278592 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.278517    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f3f59fa-913b-41d5-869c-687d01b81f4c" containerName="main"
Jul 09 17:21:55.281376 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.281359    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.285258 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.285232    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-034dd4\"/\"openshift-service-ca.crt\""
Jul 09 17:21:55.285258 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.285255    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-034dd4\"/\"default-dockercfg-cflj5\""
Jul 09 17:21:55.285438 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.285232    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-034dd4\"/\"test-model-034dd4-kserve-self-signed-certs\""
Jul 09 17:21:55.285438 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.285232    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-034dd4\"/\"kube-root-ca.crt\""
Jul 09 17:21:55.291303 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.291282    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"]
Jul 09 17:21:55.314700 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.314671    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-dshm\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.314844 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.314708    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff020a09-ce77-4344-b9c1-68d4c7af0510-tls-certs\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.314844 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.314731    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-home\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.314844 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.314793    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zsbvv\" (UniqueName: \"kubernetes.io/projected/ff020a09-ce77-4344-b9c1-68d4c7af0510-kube-api-access-zsbvv\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.314844 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.314820    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-tmp-dir\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.314997 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.314862    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-model-cache\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415260 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415217    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-model-cache\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415260 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-dshm\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415534 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415297    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff020a09-ce77-4344-b9c1-68d4c7af0510-tls-certs\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415534 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415314    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-home\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415534 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415353    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zsbvv\" (UniqueName: \"kubernetes.io/projected/ff020a09-ce77-4344-b9c1-68d4c7af0510-kube-api-access-zsbvv\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415534 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415374    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-tmp-dir\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415778 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415716    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-model-cache\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415778 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-tmp-dir\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.415842 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.415771    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-home\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.417754 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.417728    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-dshm\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.417904 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.417886    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff020a09-ce77-4344-b9c1-68d4c7af0510-tls-certs\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.423636 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.423618    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zsbvv\" (UniqueName: \"kubernetes.io/projected/ff020a09-ce77-4344-b9c1-68d4c7af0510-kube-api-access-zsbvv\") pod \"test-model-034dd4-kserve-6cbf67987-96v9x\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") " pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.584056 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.583981    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:55.590935 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.590905    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:21:55.720854 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.720830    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"]
Jul 09 17:21:55.723047 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:21:55.723020    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podff020a09_ce77_4344_b9c1_68d4c7af0510.slice/crio-a0e4c83d75183a8a93b96a75d719d19b064211519afbef2f84882ed230ed6169 WatchSource:0}: Error finding container a0e4c83d75183a8a93b96a75d719d19b064211519afbef2f84882ed230ed6169: Status 404 returned error can't find the container with id a0e4c83d75183a8a93b96a75d719d19b064211519afbef2f84882ed230ed6169
Jul 09 17:21:55.812926 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.812892    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x" event={"ID":"ff020a09-ce77-4344-b9c1-68d4c7af0510","Type":"ContainerStarted","Data":"8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52"}
Jul 09 17:21:55.813079 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.812939    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x" event={"ID":"ff020a09-ce77-4344-b9c1-68d4c7af0510","Type":"ContainerStarted","Data":"a0e4c83d75183a8a93b96a75d719d19b064211519afbef2f84882ed230ed6169"}
Jul 09 17:21:55.829889 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:55.829836    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x" podStartSLOduration=0.829818827 podStartE2EDuration="829.818827ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:21:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:21:55.828560648 +0000 UTC m=+1862.908501653" watchObservedRunningTime="2026-07-09 17:21:55.829818827 +0000 UTC m=+1862.909759847"
Jul 09 17:21:58.145809 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:58.145771    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:21:58.245230 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:21:58.245202    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:05.591028 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:05.590977    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:22:05.591483 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:05.591041    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:22:05.603735 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:05.603710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:22:05.864637 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:05.864610    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:22:06.178436 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:06.178324    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:06.276339 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:06.276308    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:20.887064 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.886974    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"]
Jul 09 17:22:20.890437 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.890418    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:20.893156 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.893132    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-90e561\"/\"openshift-service-ca.crt\""
Jul 09 17:22:20.893156 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.893149    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-90e561\"/\"kube-root-ca.crt\""
Jul 09 17:22:20.894199 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.894176    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-90e561\"/\"default-dockercfg-z496w\""
Jul 09 17:22:20.894307 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.894212    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-90e561\"/\"test-model-90e561-kserve-self-signed-certs\""
Jul 09 17:22:20.903780 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.903759    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"]
Jul 09 17:22:20.928394 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:20.928370    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:21.037307 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.037270    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5108cee-cd36-477f-86cf-141b4900b375-tls-certs\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.037476 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.037314    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-tmp-dir\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.037476 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.037381    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-model-cache\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.037476 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.037454    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-dshm\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.037581 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.037481    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-home\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.037581 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.037524    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skrrx\" (UniqueName: \"kubernetes.io/projected/b5108cee-cd36-477f-86cf-141b4900b375-kube-api-access-skrrx\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.138606 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.138518    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-model-cache\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.138606 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.138565    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-dshm\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.138606 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.138589    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-home\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.138850 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.138617    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skrrx\" (UniqueName: \"kubernetes.io/projected/b5108cee-cd36-477f-86cf-141b4900b375-kube-api-access-skrrx\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.138850 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.138643    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5108cee-cd36-477f-86cf-141b4900b375-tls-certs\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.138850 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.138702    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-tmp-dir\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.139094 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.139057    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-home\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.139094 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.139075    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-tmp-dir\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.139238 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.139107    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-model-cache\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.140845 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.140824    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-dshm\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.141088 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.141070    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5108cee-cd36-477f-86cf-141b4900b375-tls-certs\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.147105 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.147078    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-skrrx\" (UniqueName: \"kubernetes.io/projected/b5108cee-cd36-477f-86cf-141b4900b375-kube-api-access-skrrx\") pod \"test-model-90e561-kserve-7468cf4689-8mn9g\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") " pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.199779 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.199740    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:21.326678 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.326638    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"]
Jul 09 17:22:21.328708 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:22:21.328678    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb5108cee_cd36_477f_86cf_141b4900b375.slice/crio-159699f7514e68dd14271389277076b2891303ff618294fa9f72130e2a89aa39 WatchSource:0}: Error finding container 159699f7514e68dd14271389277076b2891303ff618294fa9f72130e2a89aa39: Status 404 returned error can't find the container with id 159699f7514e68dd14271389277076b2891303ff618294fa9f72130e2a89aa39
Jul 09 17:22:21.921532 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.921488    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g" event={"ID":"b5108cee-cd36-477f-86cf-141b4900b375","Type":"ContainerStarted","Data":"69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe"}
Jul 09 17:22:21.921532 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.921535    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g" event={"ID":"b5108cee-cd36-477f-86cf-141b4900b375","Type":"ContainerStarted","Data":"159699f7514e68dd14271389277076b2891303ff618294fa9f72130e2a89aa39"}
Jul 09 17:22:21.938224 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:21.938128    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g" podStartSLOduration=1.9381097999999999 podStartE2EDuration="1.9381098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:22:21.937933266 +0000 UTC m=+1889.017874234" watchObservedRunningTime="2026-07-09 17:22:21.9381098 +0000 UTC m=+1889.018050771"
Jul 09 17:22:31.200724 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:31.200691    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:31.201151 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:31.200735    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:31.213458 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:31.213434    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:31.971634 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:31.971598    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:42.279328 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:42.279293    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"]
Jul 09 17:22:42.302252 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:42.302209    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:43.333000 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:43.332972    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"]
Jul 09 17:22:43.359065 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:43.359029    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:43.532197 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:43.532163    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:44.018607 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:44.018566    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:44.120280 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:44.120246    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:44.189521 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:44.189489    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:48.987812 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:48.987775    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:22:57.299715 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.299676    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x" podUID="ff020a09-ce77-4344-b9c1-68d4c7af0510" containerName="main" containerID="cri-o://8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52" gracePeriod=45
Jul 09 17:22:57.535981 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.535955    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:22:57.653576 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653545    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-home\") pod \"ff020a09-ce77-4344-b9c1-68d4c7af0510\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") "
Jul 09 17:22:57.653779 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653591    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zsbvv\" (UniqueName: \"kubernetes.io/projected/ff020a09-ce77-4344-b9c1-68d4c7af0510-kube-api-access-zsbvv\") pod \"ff020a09-ce77-4344-b9c1-68d4c7af0510\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") "
Jul 09 17:22:57.653779 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653612    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-dshm\") pod \"ff020a09-ce77-4344-b9c1-68d4c7af0510\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") "
Jul 09 17:22:57.653779 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653629    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-model-cache\") pod \"ff020a09-ce77-4344-b9c1-68d4c7af0510\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") "
Jul 09 17:22:57.653779 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653715    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-tmp-dir\") pod \"ff020a09-ce77-4344-b9c1-68d4c7af0510\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") "
Jul 09 17:22:57.653779 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff020a09-ce77-4344-b9c1-68d4c7af0510-tls-certs\") pod \"ff020a09-ce77-4344-b9c1-68d4c7af0510\" (UID: \"ff020a09-ce77-4344-b9c1-68d4c7af0510\") "
Jul 09 17:22:57.654020 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653900    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-home" (OuterVolumeSpecName: "home") pod "ff020a09-ce77-4344-b9c1-68d4c7af0510" (UID: "ff020a09-ce77-4344-b9c1-68d4c7af0510"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:57.654020 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653939    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-model-cache" (OuterVolumeSpecName: "model-cache") pod "ff020a09-ce77-4344-b9c1-68d4c7af0510" (UID: "ff020a09-ce77-4344-b9c1-68d4c7af0510"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:57.654020 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.653954    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ff020a09-ce77-4344-b9c1-68d4c7af0510" (UID: "ff020a09-ce77-4344-b9c1-68d4c7af0510"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:57.655915 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.655887    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-dshm" (OuterVolumeSpecName: "dshm") pod "ff020a09-ce77-4344-b9c1-68d4c7af0510" (UID: "ff020a09-ce77-4344-b9c1-68d4c7af0510"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:57.656024 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.655949    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ff020a09-ce77-4344-b9c1-68d4c7af0510-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ff020a09-ce77-4344-b9c1-68d4c7af0510" (UID: "ff020a09-ce77-4344-b9c1-68d4c7af0510"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:22:57.656024 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.655945    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ff020a09-ce77-4344-b9c1-68d4c7af0510-kube-api-access-zsbvv" (OuterVolumeSpecName: "kube-api-access-zsbvv") pod "ff020a09-ce77-4344-b9c1-68d4c7af0510" (UID: "ff020a09-ce77-4344-b9c1-68d4c7af0510"). InnerVolumeSpecName "kube-api-access-zsbvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:22:57.754782 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.754746    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:57.754782 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.754778    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ff020a09-ce77-4344-b9c1-68d4c7af0510-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:57.754964 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.754792    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:57.754964 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.754808    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zsbvv\" (UniqueName: \"kubernetes.io/projected/ff020a09-ce77-4344-b9c1-68d4c7af0510-kube-api-access-zsbvv\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:57.754964 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.754821    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:57.754964 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:57.754836    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ff020a09-ce77-4344-b9c1-68d4c7af0510-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:58.068355 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.068266    2563 generic.go:356] "Generic (PLEG): container finished" podID="ff020a09-ce77-4344-b9c1-68d4c7af0510" containerID="8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52" exitCode=0
Jul 09 17:22:58.068355 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.068308    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x" event={"ID":"ff020a09-ce77-4344-b9c1-68d4c7af0510","Type":"ContainerDied","Data":"8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52"}
Jul 09 17:22:58.068355 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.068316    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"
Jul 09 17:22:58.068355 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.068346    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x" event={"ID":"ff020a09-ce77-4344-b9c1-68d4c7af0510","Type":"ContainerDied","Data":"a0e4c83d75183a8a93b96a75d719d19b064211519afbef2f84882ed230ed6169"}
Jul 09 17:22:58.068597 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.068364    2563 scope.go:117] "RemoveContainer" containerID="8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52"
Jul 09 17:22:58.077335 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.077319    2563 scope.go:117] "RemoveContainer" containerID="8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52"
Jul 09 17:22:58.077586 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:22:58.077568    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52\": container with ID starting with 8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52 not found: ID does not exist" containerID="8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52"
Jul 09 17:22:58.077623 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.077595    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52"} err="failed to get container status \"8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52\": rpc error: code = NotFound desc = could not find container \"8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52\": container with ID starting with 8e7f2a14788402fdb5d306dd73c9afa31ce8475a45ef0a7645fbfbcc07189d52 not found: ID does not exist"
Jul 09 17:22:58.089780 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.089756    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"]
Jul 09 17:22:58.093572 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.093550    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-034dd4/test-model-034dd4-kserve-6cbf67987-96v9x"]
Jul 09 17:22:58.355872 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.355779    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g" podUID="b5108cee-cd36-477f-86cf-141b4900b375" containerName="main" containerID="cri-o://69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe" gracePeriod=45
Jul 09 17:22:58.611194 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.611173    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:58.664089 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664055    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-skrrx\" (UniqueName: \"kubernetes.io/projected/b5108cee-cd36-477f-86cf-141b4900b375-kube-api-access-skrrx\") pod \"b5108cee-cd36-477f-86cf-141b4900b375\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") "
Jul 09 17:22:58.664255 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664120    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-tmp-dir\") pod \"b5108cee-cd36-477f-86cf-141b4900b375\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") "
Jul 09 17:22:58.664255 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664147    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5108cee-cd36-477f-86cf-141b4900b375-tls-certs\") pod \"b5108cee-cd36-477f-86cf-141b4900b375\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") "
Jul 09 17:22:58.664255 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664174    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-model-cache\") pod \"b5108cee-cd36-477f-86cf-141b4900b375\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") "
Jul 09 17:22:58.664255 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664228    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-home\") pod \"b5108cee-cd36-477f-86cf-141b4900b375\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") "
Jul 09 17:22:58.664444 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664281    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-dshm\") pod \"b5108cee-cd36-477f-86cf-141b4900b375\" (UID: \"b5108cee-cd36-477f-86cf-141b4900b375\") "
Jul 09 17:22:58.664496 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664431    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b5108cee-cd36-477f-86cf-141b4900b375" (UID: "b5108cee-cd36-477f-86cf-141b4900b375"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:58.664496 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-model-cache" (OuterVolumeSpecName: "model-cache") pod "b5108cee-cd36-477f-86cf-141b4900b375" (UID: "b5108cee-cd36-477f-86cf-141b4900b375"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:58.664594 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664547    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-home" (OuterVolumeSpecName: "home") pod "b5108cee-cd36-477f-86cf-141b4900b375" (UID: "b5108cee-cd36-477f-86cf-141b4900b375"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:58.664594 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664578    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:58.664698 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.664597    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:58.666446 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.666416    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5108cee-cd36-477f-86cf-141b4900b375-kube-api-access-skrrx" (OuterVolumeSpecName: "kube-api-access-skrrx") pod "b5108cee-cd36-477f-86cf-141b4900b375" (UID: "b5108cee-cd36-477f-86cf-141b4900b375"). InnerVolumeSpecName "kube-api-access-skrrx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:22:58.666446 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.666419    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b5108cee-cd36-477f-86cf-141b4900b375-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b5108cee-cd36-477f-86cf-141b4900b375" (UID: "b5108cee-cd36-477f-86cf-141b4900b375"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:22:58.666591 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.666419    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-dshm" (OuterVolumeSpecName: "dshm") pod "b5108cee-cd36-477f-86cf-141b4900b375" (UID: "b5108cee-cd36-477f-86cf-141b4900b375"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:22:58.765150 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.765116    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-skrrx\" (UniqueName: \"kubernetes.io/projected/b5108cee-cd36-477f-86cf-141b4900b375-kube-api-access-skrrx\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:58.765150 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.765146    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b5108cee-cd36-477f-86cf-141b4900b375-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:58.765150 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.765156    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:58.765375 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:58.765164    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b5108cee-cd36-477f-86cf-141b4900b375-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:22:59.075162 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.075068    2563 generic.go:356] "Generic (PLEG): container finished" podID="b5108cee-cd36-477f-86cf-141b4900b375" containerID="69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe" exitCode=0
Jul 09 17:22:59.075162 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.075108    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"
Jul 09 17:22:59.075162 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.075125    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g" event={"ID":"b5108cee-cd36-477f-86cf-141b4900b375","Type":"ContainerDied","Data":"69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe"}
Jul 09 17:22:59.075162 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.075157    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g" event={"ID":"b5108cee-cd36-477f-86cf-141b4900b375","Type":"ContainerDied","Data":"159699f7514e68dd14271389277076b2891303ff618294fa9f72130e2a89aa39"}
Jul 09 17:22:59.075466 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.075173    2563 scope.go:117] "RemoveContainer" containerID="69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe"
Jul 09 17:22:59.083633 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.083616    2563 scope.go:117] "RemoveContainer" containerID="69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe"
Jul 09 17:22:59.083991 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:22:59.083970    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe\": container with ID starting with 69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe not found: ID does not exist" containerID="69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe"
Jul 09 17:22:59.084043 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.083999    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe"} err="failed to get container status \"69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe\": rpc error: code = NotFound desc = could not find container \"69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe\": container with ID starting with 69ad809c3d06baba45dfd8970e3e4a89997bb7907b0056142290d9ddff33febe not found: ID does not exist"
Jul 09 17:22:59.095590 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.095560    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"]
Jul 09 17:22:59.099151 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.099130    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-90e561/test-model-90e561-kserve-7468cf4689-8mn9g"]
Jul 09 17:22:59.353300 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.353220    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5108cee-cd36-477f-86cf-141b4900b375" path="/var/lib/kubelet/pods/b5108cee-cd36-477f-86cf-141b4900b375/volumes"
Jul 09 17:22:59.353618 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:22:59.353605    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ff020a09-ce77-4344-b9c1-68d4c7af0510" path="/var/lib/kubelet/pods/ff020a09-ce77-4344-b9c1-68d4c7af0510/volumes"
Jul 09 17:23:07.417283 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:07.417254    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:26.133243 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:26.133206    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:27.771647 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:27.771610    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:33.008224 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:33.008195    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:33.106093 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:33.106063    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:33.279840 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:33.279755    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:33.479172 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:33.479137    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:33.834204 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:33.834175    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:34.181958 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:34.181924    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:34.389415 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:34.389381    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:37.431604 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:37.431562    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:37.519774 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:37.519732    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:37.583103 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:37.583071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:37.785818 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:37.785730    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:38.000902 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:38.000873    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:38.101915 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:38.101831    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:41.953268 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:41.953235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:42.433212 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:42.433178    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:42.735772 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:42.735694    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:42.991758 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:42.991670    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:45.331528 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.331445    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:45.425049 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.425015    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:45.523857 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.523826    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"]
Jul 09 17:23:45.524127 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.524115    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5108cee-cd36-477f-86cf-141b4900b375" containerName="main"
Jul 09 17:23:45.524189 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.524128    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5108cee-cd36-477f-86cf-141b4900b375" containerName="main"
Jul 09 17:23:45.524189 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.524139    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff020a09-ce77-4344-b9c1-68d4c7af0510" containerName="main"
Jul 09 17:23:45.524189 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.524145    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff020a09-ce77-4344-b9c1-68d4c7af0510" containerName="main"
Jul 09 17:23:45.524335 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.524200    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5108cee-cd36-477f-86cf-141b4900b375" containerName="main"
Jul 09 17:23:45.524335 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.524213    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff020a09-ce77-4344-b9c1-68d4c7af0510" containerName="main"
Jul 09 17:23:45.527121 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.527100    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.529856 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.529827    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c494fe-serving-cert\""
Jul 09 17:23:45.530000 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.529983    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:23:45.530060 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.530053    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:23:45.531200 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.531185    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mpbsc\""
Jul 09 17:23:45.541208 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.541189    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"]
Jul 09 17:23:45.546878 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.546857    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.546988 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.546911    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tp9fm\" (UniqueName: \"kubernetes.io/projected/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-kube-api-access-tp9fm\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.546988 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.546945    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-openshift-service-ca\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.565675 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.565629    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:45.648031 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.647997    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.648217 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.648044    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tp9fm\" (UniqueName: \"kubernetes.io/projected/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-kube-api-access-tp9fm\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.648217 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.648144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-openshift-service-ca\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.648217 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:23:45.648166    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-serving-cert: secret "maas-api-e2e-api-b-c494fe-serving-cert" not found
Jul 09 17:23:45.648336 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:23:45.648258    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls podName:62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39 nodeName:}" failed. No retries permitted until 2026-07-09 17:23:46.148236116 +0000 UTC m=+1973.228177068 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls") pod "maas-api-e2e-api-b-c494fe-55cbb98944-bl72w" (UID: "62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39") : secret "maas-api-e2e-api-b-c494fe-serving-cert" not found
Jul 09 17:23:45.648820 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.648801    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-openshift-service-ca\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:45.656532 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:45.656503    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tp9fm\" (UniqueName: \"kubernetes.io/projected/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-kube-api-access-tp9fm\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:46.006173 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.006084    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:46.105953 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.105920    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:46.151211 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.151176    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:46.153558 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.153527    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls\") pod \"maas-api-e2e-api-b-c494fe-55cbb98944-bl72w\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:46.215138 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.215110    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:46.438822 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.438784    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:46.567542 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.567499    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"]
Jul 09 17:23:46.569696 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:23:46.569669    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62381c9c_9f2d_4eb5_b9cf_2a5c64e68f39.slice/crio-df9124124c2feebcd0077dc9eb5691fafbdde74f4b2cdc94eb9726ea148a79f5 WatchSource:0}: Error finding container df9124124c2feebcd0077dc9eb5691fafbdde74f4b2cdc94eb9726ea148a79f5: Status 404 returned error can't find the container with id df9124124c2feebcd0077dc9eb5691fafbdde74f4b2cdc94eb9726ea148a79f5
Jul 09 17:23:46.571012 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:46.570996    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:23:47.269042 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:47.269006    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w" event={"ID":"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39","Type":"ContainerStarted","Data":"df9124124c2feebcd0077dc9eb5691fafbdde74f4b2cdc94eb9726ea148a79f5"}
Jul 09 17:23:49.280322 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:49.280281    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w" event={"ID":"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39","Type":"ContainerStarted","Data":"e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6"}
Jul 09 17:23:49.280770 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:49.280397    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:49.298239 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:49.298190    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w" podStartSLOduration=2.136222404 podStartE2EDuration="4.298173175s" totalImagesPullingTime="2.161950771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:23:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:23:49.296315907 +0000 UTC m=+1976.376256889" watchObservedRunningTime="2026-07-09 17:23:49.298173175 +0000 UTC m=+1976.378114145"
Jul 09 17:23:49.723862 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:49.723835    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:55.289309 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:55.289277    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:23:57.718266 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:57.718235    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:58.100946 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:58.100870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:58.203219 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:58.203167    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:58.302168 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:58.302136    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:23:58.449822 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:23:58.449790    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:03.993829 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:03.993798    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:04.094191 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:04.094158    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:04.261993 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:04.261899    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:04.466592 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:04.466564    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:04.706089 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:04.706059    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:05.050178 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.050105    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:05.153492 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.153461    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:05.251100 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.251071    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:05.401503 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.401473    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"]
Jul 09 17:24:05.401698 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.401677    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w" podUID="62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" containerName="maas-api" containerID="cri-o://e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6" gracePeriod=30
Jul 09 17:24:05.429166 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.429135    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:05.637184 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.637161    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:24:05.727874 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.727801    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-openshift-service-ca\") pod \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") "
Jul 09 17:24:05.727874 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.727842    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tp9fm\" (UniqueName: \"kubernetes.io/projected/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-kube-api-access-tp9fm\") pod \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") "
Jul 09 17:24:05.727874 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.727866    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls\") pod \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\" (UID: \"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39\") "
Jul 09 17:24:05.728145 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.728113    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" (UID: "62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:24:05.730037 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.729996    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" (UID: "62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:24:05.730144 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.730050    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-kube-api-access-tp9fm" (OuterVolumeSpecName: "kube-api-access-tp9fm") pod "62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" (UID: "62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39"). InnerVolumeSpecName "kube-api-access-tp9fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:24:05.829243 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.829215    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tp9fm\" (UniqueName: \"kubernetes.io/projected/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-kube-api-access-tp9fm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:24:05.829243 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.829240    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:24:05.829243 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:05.829249    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:24:06.349501 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.349465    2563 generic.go:356] "Generic (PLEG): container finished" podID="62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" containerID="e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6" exitCode=0
Jul 09 17:24:06.349971 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.349510    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"
Jul 09 17:24:06.349971 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.349525    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w" event={"ID":"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39","Type":"ContainerDied","Data":"e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6"}
Jul 09 17:24:06.349971 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.349563    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w" event={"ID":"62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39","Type":"ContainerDied","Data":"df9124124c2feebcd0077dc9eb5691fafbdde74f4b2cdc94eb9726ea148a79f5"}
Jul 09 17:24:06.349971 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.349584    2563 scope.go:117] "RemoveContainer" containerID="e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6"
Jul 09 17:24:06.358197 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.358181    2563 scope.go:117] "RemoveContainer" containerID="e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6"
Jul 09 17:24:06.358479 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:24:06.358458    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6\": container with ID starting with e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6 not found: ID does not exist" containerID="e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6"
Jul 09 17:24:06.358517 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.358490    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6"} err="failed to get container status \"e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6\": rpc error: code = NotFound desc = could not find container \"e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6\": container with ID starting with e10f263ae56b710d8deda221d5ed006b7d14521a0ba1409bf53de648eea6cfa6 not found: ID does not exist"
Jul 09 17:24:06.371048 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.371023    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"]
Jul 09 17:24:06.375157 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:06.375135    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c494fe-55cbb98944-bl72w"]
Jul 09 17:24:07.352428 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:07.352395    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" path="/var/lib/kubelet/pods/62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39/volumes"
Jul 09 17:24:11.115439 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:11.115404    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:11.216233 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:11.216203    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:11.366492 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:11.366379    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:11.576219 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:11.576191    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:11.765495 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:11.765403    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:11.868718 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:11.868686    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:11.965980 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:11.965947    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:12.336031 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:12.335996    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:12.622510 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:12.622475    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:14.767411 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:14.767372    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:14.859244 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:14.859217    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:14.939702 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:14.939646    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:15.035531 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:15.035435    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:15.316148 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:15.316051    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:15.417691 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:15.417641    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:19.152927 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:19.152879    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:19.472122 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:19.472038    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:19.767159 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:19.767076    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:19.937807 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:19.937779    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:22.508378 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.508334    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:22.576882 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.576850    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"]
Jul 09 17:24:22.577179 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.577163    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" containerName="maas-api"
Jul 09 17:24:22.577179 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.577178    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" containerName="maas-api"
Jul 09 17:24:22.577335 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.577245    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="62381c9c-9f2d-4eb5-b9cf-2a5c64e68f39" containerName="maas-api"
Jul 09 17:24:22.585718 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.585696    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.588748 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.588724    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 17:24:22.588879 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.588826    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 17:24:22.589941 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.589919    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-ba9b25-serving-cert\""
Jul 09 17:24:22.590052 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.589922    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-mpbsc\""
Jul 09 17:24:22.595609 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.595555    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:22.597664 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.597629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"]
Jul 09 17:24:22.665940 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.665910    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:22.672017 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.671994    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/5b9e7f5d-3bc9-47a4-9764-275c63091356-kube-api-access-6wsgr\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.672126 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.672033    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.672126 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.672105    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e7f5d-3bc9-47a4-9764-275c63091356-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.772849 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.772771    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/5b9e7f5d-3bc9-47a4-9764-275c63091356-kube-api-access-6wsgr\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.772849 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.772808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.772849 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.772848    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e7f5d-3bc9-47a4-9764-275c63091356-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.773063 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:24:22.772984    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-serving-cert: secret "maas-api-e2e-shared-b-ba9b25-serving-cert" not found
Jul 09 17:24:22.773109 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:24:22.773064    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls podName:5b9e7f5d-3bc9-47a4-9764-275c63091356 nodeName:}" failed. No retries permitted until 2026-07-09 17:24:23.273041716 +0000 UTC m=+2010.352982677 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls") pod "maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" (UID: "5b9e7f5d-3bc9-47a4-9764-275c63091356") : secret "maas-api-e2e-shared-b-ba9b25-serving-cert" not found
Jul 09 17:24:22.773558 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.773540    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e7f5d-3bc9-47a4-9764-275c63091356-openshift-service-ca\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:22.781784 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:22.781760    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/5b9e7f5d-3bc9-47a4-9764-275c63091356-kube-api-access-6wsgr\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:23.180319 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:23.180284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:23.277110 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:23.277070    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:23.277270 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:24:23.277228    2563 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-serving-cert: secret "maas-api-e2e-shared-b-ba9b25-serving-cert" not found
Jul 09 17:24:23.277314 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:24:23.277292    2563 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls podName:5b9e7f5d-3bc9-47a4-9764-275c63091356 nodeName:}" failed. No retries permitted until 2026-07-09 17:24:24.277276387 +0000 UTC m=+2011.357217338 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls") pod "maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" (UID: "5b9e7f5d-3bc9-47a4-9764-275c63091356") : secret "maas-api-e2e-shared-b-ba9b25-serving-cert" not found
Jul 09 17:24:23.283860 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:23.283834    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:23.380854 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:23.380828    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:24.284309 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:24.284277    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:24.286672 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:24.286634    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls\") pod \"maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:24.399907 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:24.399877    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:24.527994 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:24.527971    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"]
Jul 09 17:24:24.530447 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:24:24.530417    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b9e7f5d_3bc9_47a4_9764_275c63091356.slice/crio-370414ef1be4f320f8272f22be99a972cb9fb0c29cb47340e4e65e2e256c428c WatchSource:0}: Error finding container 370414ef1be4f320f8272f22be99a972cb9fb0c29cb47340e4e65e2e256c428c: Status 404 returned error can't find the container with id 370414ef1be4f320f8272f22be99a972cb9fb0c29cb47340e4e65e2e256c428c
Jul 09 17:24:25.426472 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:25.426438    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" event={"ID":"5b9e7f5d-3bc9-47a4-9764-275c63091356","Type":"ContainerStarted","Data":"c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce"}
Jul 09 17:24:25.426855 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:25.426487    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" event={"ID":"5b9e7f5d-3bc9-47a4-9764-275c63091356","Type":"ContainerStarted","Data":"370414ef1be4f320f8272f22be99a972cb9fb0c29cb47340e4e65e2e256c428c"}
Jul 09 17:24:25.426855 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:25.426524    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:25.444101 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:25.444059    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" podStartSLOduration=2.98271368 podStartE2EDuration="3.444044333s" totalImagesPullingTime="461.330653ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:24:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:24:25.443299506 +0000 UTC m=+2012.523240457" watchObservedRunningTime="2026-07-09 17:24:25.444044333 +0000 UTC m=+2012.523985302"
Jul 09 17:24:27.751011 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:27.750965    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:31.436500 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:31.436466    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:24:36.195007 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:36.194959    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:36.248900 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:36.248870    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:37.592275 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.592242    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"]
Jul 09 17:24:37.595682 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.595662    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.598634 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.598615    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"kube-root-ca.crt\""
Jul 09 17:24:37.599628 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.599611    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"auth-test-model-553a9f-kserve-self-signed-certs\""
Jul 09 17:24:37.599628 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.599619    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"openshift-service-ca.crt\""
Jul 09 17:24:37.599839 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.599821    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"default-dockercfg-46p5h\""
Jul 09 17:24:37.603436 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.603415    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"]
Jul 09 17:24:37.618183 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.618162    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:37.696598 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.696567    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-model-cache\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.696797 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.696620    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tls-certs\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.696797 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.696722    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-dshm\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.696797 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.696749    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tmp-dir\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.696797 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.696779    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4cd8\" (UniqueName: \"kubernetes.io/projected/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-kube-api-access-n4cd8\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.696932 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.696827    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-home\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.797837 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.797808    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-model-cache\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.797965 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.797857    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tls-certs\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.797965 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.797900    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-dshm\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.797965 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.797926    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tmp-dir\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.798094 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.797961    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4cd8\" (UniqueName: \"kubernetes.io/projected/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-kube-api-access-n4cd8\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.798094 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.797991    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-home\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.798301 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.798281    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-model-cache\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.798370 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.798321    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-home\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.798469 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.798447    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tmp-dir\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.800215 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.800192    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-dshm\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.800475 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.800456    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tls-certs\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.806413 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.806388    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4cd8\" (UniqueName: \"kubernetes.io/projected/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-kube-api-access-n4cd8\") pod \"auth-test-model-553a9f-kserve-695d8d6c55-9jf9x\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") " pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:37.904566 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:37.904531    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:38.029433 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:38.029409    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"]
Jul 09 17:24:38.031073 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:24:38.031046    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode4d6052f_e157_42ac_91bd_4b11a5ccbc9f.slice/crio-c64a43adbcc4307653af9915e13ce58fd16a668f066041d9ce8847ee46c56f5c WatchSource:0}: Error finding container c64a43adbcc4307653af9915e13ce58fd16a668f066041d9ce8847ee46c56f5c: Status 404 returned error can't find the container with id c64a43adbcc4307653af9915e13ce58fd16a668f066041d9ce8847ee46c56f5c
Jul 09 17:24:38.481690 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:38.481633    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x" event={"ID":"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f","Type":"ContainerStarted","Data":"51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad"}
Jul 09 17:24:38.481853 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:38.481701    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x" event={"ID":"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f","Type":"ContainerStarted","Data":"c64a43adbcc4307653af9915e13ce58fd16a668f066041d9ce8847ee46c56f5c"}
Jul 09 17:24:38.501002 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:38.500956    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x" podStartSLOduration=1.500941767 podStartE2EDuration="1.500941767s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:24:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:24:38.499354906 +0000 UTC m=+2025.579295873" watchObservedRunningTime="2026-07-09 17:24:38.500941767 +0000 UTC m=+2025.580882735"
Jul 09 17:24:44.945336 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:44.945263    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:45.045974 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:45.045938    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:24:47.905296 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:47.905252    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:47.905701 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:47.905310    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:47.917393 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:47.917362    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:24:48.534065 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:24:48.534036    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:25:10.403838 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.403799    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"]
Jul 09 17:25:10.406851 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.406835    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.410785 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.410758    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"default-dockercfg-9gd5z\""
Jul 09 17:25:10.410930 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.410902    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"openshift-service-ca.crt\""
Jul 09 17:25:10.410930 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.410912    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"kube-root-ca.crt\""
Jul 09 17:25:10.411045 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.411031    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"auth-test-model-ba9b25-kserve-self-signed-certs\""
Jul 09 17:25:10.422335 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.417395    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"]
Jul 09 17:25:10.483506 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.483478    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-home\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.483678 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.483518    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf5f480-dbf1-4521-b3ba-9f31736c723b-tls-certs\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.483678 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.483578    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dm7sd\" (UniqueName: \"kubernetes.io/projected/caf5f480-dbf1-4521-b3ba-9f31736c723b-kube-api-access-dm7sd\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.483765 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.483688    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-dshm\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.483810 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.483786    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-model-cache\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.483875 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.483861    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-tmp-dir\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.510957 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.510925    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:25:10.585129 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dm7sd\" (UniqueName: \"kubernetes.io/projected/caf5f480-dbf1-4521-b3ba-9f31736c723b-kube-api-access-dm7sd\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585144    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-dshm\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585194    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-model-cache\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585236    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-tmp-dir\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585279    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-home\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf5f480-dbf1-4521-b3ba-9f31736c723b-tls-certs\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585711 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585686    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-model-cache\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585788 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585682    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-tmp-dir\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.585788 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.585732    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-home\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.587570 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.587550    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-dshm\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.587871 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.587853    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf5f480-dbf1-4521-b3ba-9f31736c723b-tls-certs\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.593357 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.593336    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dm7sd\" (UniqueName: \"kubernetes.io/projected/caf5f480-dbf1-4521-b3ba-9f31736c723b-kube-api-access-dm7sd\") pod \"auth-test-model-ba9b25-kserve-5dd844b995-77j8f\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") " pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.717518 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.717427    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:10.850927 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:10.850901    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"]
Jul 09 17:25:10.853399 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:25:10.853374    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podcaf5f480_dbf1_4521_b3ba_9f31736c723b.slice/crio-bf32ed8de483f2f263e73663e0820475c160fe7622c7eae9e6c1d42f3eb0fd34 WatchSource:0}: Error finding container bf32ed8de483f2f263e73663e0820475c160fe7622c7eae9e6c1d42f3eb0fd34: Status 404 returned error can't find the container with id bf32ed8de483f2f263e73663e0820475c160fe7622c7eae9e6c1d42f3eb0fd34
Jul 09 17:25:11.615430 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:11.615399    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f" event={"ID":"caf5f480-dbf1-4521-b3ba-9f31736c723b","Type":"ContainerStarted","Data":"14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7"}
Jul 09 17:25:11.615833 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:11.615444    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f" event={"ID":"caf5f480-dbf1-4521-b3ba-9f31736c723b","Type":"ContainerStarted","Data":"bf32ed8de483f2f263e73663e0820475c160fe7622c7eae9e6c1d42f3eb0fd34"}
Jul 09 17:25:11.631735 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:11.631641    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f" podStartSLOduration=1.631618976 podStartE2EDuration="1.631618976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:25:11.630065209 +0000 UTC m=+2058.710006204" watchObservedRunningTime="2026-07-09 17:25:11.631618976 +0000 UTC m=+2058.711559951"
Jul 09 17:25:20.717793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:20.717710    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:20.717793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:20.717764    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:20.730163 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:20.730145    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:25:21.667870 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:25:21.667841    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:26:19.303532 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:26:19.303500    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:26:19.358642 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:26:19.358614    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:26:44.606384 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:26:44.606354    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:26:52.129099 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:26:52.129016    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:27:15.749950 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:15.749912    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"]
Jul 09 17:27:15.768666 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:15.768622    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:27:16.584360 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:16.584327    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"]
Jul 09 17:27:16.610753 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:16.610723    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:27:28.944256 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:28.944225    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:27:30.770240 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:30.770183    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x" podUID="e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" containerName="main" containerID="cri-o://51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad" gracePeriod=45
Jul 09 17:27:31.021382 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.021313    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:27:31.094787 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.094753    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-dshm\") pod \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") "
Jul 09 17:27:31.094787 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.094792    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4cd8\" (UniqueName: \"kubernetes.io/projected/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-kube-api-access-n4cd8\") pod \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") "
Jul 09 17:27:31.095040 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.094835    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tmp-dir\") pod \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") "
Jul 09 17:27:31.095040 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.094861    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tls-certs\") pod \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") "
Jul 09 17:27:31.095040 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.094906    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-model-cache\") pod \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") "
Jul 09 17:27:31.095040 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.094937    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-home\") pod \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\" (UID: \"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f\") "
Jul 09 17:27:31.095262 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.095135    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" (UID: "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:31.095262 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.095199    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-model-cache" (OuterVolumeSpecName: "model-cache") pod "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" (UID: "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:31.095351 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.095305    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-home" (OuterVolumeSpecName: "home") pod "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" (UID: "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:31.097221 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.097196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-dshm" (OuterVolumeSpecName: "dshm") pod "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" (UID: "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:31.097221 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.097196    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" (UID: "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:27:31.097344 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.097253    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-kube-api-access-n4cd8" (OuterVolumeSpecName: "kube-api-access-n4cd8") pod "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" (UID: "e4d6052f-e157-42ac-91bd-4b11a5ccbc9f"). InnerVolumeSpecName "kube-api-access-n4cd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:27:31.185043 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.185011    2563 generic.go:356] "Generic (PLEG): container finished" podID="e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" containerID="51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad" exitCode=0
Jul 09 17:27:31.185212 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.185049    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"
Jul 09 17:27:31.185212 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.185098    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x" event={"ID":"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f","Type":"ContainerDied","Data":"51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad"}
Jul 09 17:27:31.185212 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.185140    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x" event={"ID":"e4d6052f-e157-42ac-91bd-4b11a5ccbc9f","Type":"ContainerDied","Data":"c64a43adbcc4307653af9915e13ce58fd16a668f066041d9ce8847ee46c56f5c"}
Jul 09 17:27:31.185212 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.185160    2563 scope.go:117] "RemoveContainer" containerID="51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad"
Jul 09 17:27:31.194126 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.194104    2563 scope.go:117] "RemoveContainer" containerID="51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad"
Jul 09 17:27:31.194374 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:27:31.194353    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad\": container with ID starting with 51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad not found: ID does not exist" containerID="51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad"
Jul 09 17:27:31.194441 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.194381    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad"} err="failed to get container status \"51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad\": rpc error: code = NotFound desc = could not find container \"51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad\": container with ID starting with 51e34a85cb68269cb789809025f9c6a53eea25baa914df6a96bb9d93b1df3dad not found: ID does not exist"
Jul 09 17:27:31.195740 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.195725    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:31.195793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.195752    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:31.195793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.195762    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:31.195793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.195770    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:31.195793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.195778    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:31.195793 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.195785    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4cd8\" (UniqueName: \"kubernetes.io/projected/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f-kube-api-access-n4cd8\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:31.206553 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.206534    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"]
Jul 09 17:27:31.210929 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.210911    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/auth-test-model-553a9f-kserve-695d8d6c55-9jf9x"]
Jul 09 17:27:31.353075 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.352993    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" path="/var/lib/kubelet/pods/e4d6052f-e157-42ac-91bd-4b11a5ccbc9f/volumes"
Jul 09 17:27:31.605285 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.605197    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f" podUID="caf5f480-dbf1-4521-b3ba-9f31736c723b" containerName="main" containerID="cri-o://14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7" gracePeriod=45
Jul 09 17:27:31.657819 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.657786    2563 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f" podUID="caf5f480-dbf1-4521-b3ba-9f31736c723b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.36:8000/ready\": dial tcp 10.133.0.36:8000: connect: connection refused"
Jul 09 17:27:31.854489 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:31.854469    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:27:32.002741 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.002709    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dm7sd\" (UniqueName: \"kubernetes.io/projected/caf5f480-dbf1-4521-b3ba-9f31736c723b-kube-api-access-dm7sd\") pod \"caf5f480-dbf1-4521-b3ba-9f31736c723b\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") "
Jul 09 17:27:32.002953 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.002749    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-dshm\") pod \"caf5f480-dbf1-4521-b3ba-9f31736c723b\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") "
Jul 09 17:27:32.002953 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.002807    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-home\") pod \"caf5f480-dbf1-4521-b3ba-9f31736c723b\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") "
Jul 09 17:27:32.002953 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.002854    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-tmp-dir\") pod \"caf5f480-dbf1-4521-b3ba-9f31736c723b\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") "
Jul 09 17:27:32.002953 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.002884    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf5f480-dbf1-4521-b3ba-9f31736c723b-tls-certs\") pod \"caf5f480-dbf1-4521-b3ba-9f31736c723b\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") "
Jul 09 17:27:32.002953 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.002928    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-model-cache\") pod \"caf5f480-dbf1-4521-b3ba-9f31736c723b\" (UID: \"caf5f480-dbf1-4521-b3ba-9f31736c723b\") "
Jul 09 17:27:32.003212 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.003082    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-home" (OuterVolumeSpecName: "home") pod "caf5f480-dbf1-4521-b3ba-9f31736c723b" (UID: "caf5f480-dbf1-4521-b3ba-9f31736c723b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:32.003212 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.003118    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "caf5f480-dbf1-4521-b3ba-9f31736c723b" (UID: "caf5f480-dbf1-4521-b3ba-9f31736c723b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:32.003315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.003224    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-model-cache" (OuterVolumeSpecName: "model-cache") pod "caf5f480-dbf1-4521-b3ba-9f31736c723b" (UID: "caf5f480-dbf1-4521-b3ba-9f31736c723b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:32.003315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.003234    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:32.003315 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.003250    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:32.005031 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.005009    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-dshm" (OuterVolumeSpecName: "dshm") pod "caf5f480-dbf1-4521-b3ba-9f31736c723b" (UID: "caf5f480-dbf1-4521-b3ba-9f31736c723b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:27:32.005145 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.005123    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf5f480-dbf1-4521-b3ba-9f31736c723b-kube-api-access-dm7sd" (OuterVolumeSpecName: "kube-api-access-dm7sd") pod "caf5f480-dbf1-4521-b3ba-9f31736c723b" (UID: "caf5f480-dbf1-4521-b3ba-9f31736c723b"). InnerVolumeSpecName "kube-api-access-dm7sd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:27:32.005145 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.005137    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/caf5f480-dbf1-4521-b3ba-9f31736c723b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "caf5f480-dbf1-4521-b3ba-9f31736c723b" (UID: "caf5f480-dbf1-4521-b3ba-9f31736c723b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:27:32.103847 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.103817    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/caf5f480-dbf1-4521-b3ba-9f31736c723b-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:32.103847 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.103841    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:32.103847 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.103851    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dm7sd\" (UniqueName: \"kubernetes.io/projected/caf5f480-dbf1-4521-b3ba-9f31736c723b-kube-api-access-dm7sd\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:32.104059 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.103860    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/caf5f480-dbf1-4521-b3ba-9f31736c723b-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:27:32.192041 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.192006    2563 generic.go:356] "Generic (PLEG): container finished" podID="caf5f480-dbf1-4521-b3ba-9f31736c723b" containerID="14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7" exitCode=0
Jul 09 17:27:32.192206 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.192047    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"
Jul 09 17:27:32.192206 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.192053    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f" event={"ID":"caf5f480-dbf1-4521-b3ba-9f31736c723b","Type":"ContainerDied","Data":"14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7"}
Jul 09 17:27:32.192206 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.192085    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f" event={"ID":"caf5f480-dbf1-4521-b3ba-9f31736c723b","Type":"ContainerDied","Data":"bf32ed8de483f2f263e73663e0820475c160fe7622c7eae9e6c1d42f3eb0fd34"}
Jul 09 17:27:32.192206 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.192101    2563 scope.go:117] "RemoveContainer" containerID="14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7"
Jul 09 17:27:32.200381 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.200365    2563 scope.go:117] "RemoveContainer" containerID="14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7"
Jul 09 17:27:32.200633 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:27:32.200615    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7\": container with ID starting with 14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7 not found: ID does not exist" containerID="14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7"
Jul 09 17:27:32.200723 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.200641    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7"} err="failed to get container status \"14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7\": rpc error: code = NotFound desc = could not find container \"14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7\": container with ID starting with 14eececfd878ffb89ac64212c2809c6e2be5e55a116922afdadeff21af671ea7 not found: ID does not exist"
Jul 09 17:27:32.212155 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.212134    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"]
Jul 09 17:27:32.215972 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:32.215953    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/auth-test-model-ba9b25-kserve-5dd844b995-77j8f"]
Jul 09 17:27:33.357758 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:33.357721    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="caf5f480-dbf1-4521-b3ba-9f31736c723b" path="/var/lib/kubelet/pods/caf5f480-dbf1-4521-b3ba-9f31736c723b/volumes"
Jul 09 17:27:50.990196 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.990157    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"]
Jul 09 17:27:50.990683 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.990440    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="caf5f480-dbf1-4521-b3ba-9f31736c723b" containerName="main"
Jul 09 17:27:50.990683 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.990450    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="caf5f480-dbf1-4521-b3ba-9f31736c723b" containerName="main"
Jul 09 17:27:50.990683 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.990463    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" containerName="main"
Jul 09 17:27:50.990683 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.990469    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" containerName="main"
Jul 09 17:27:50.990683 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.990524    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="e4d6052f-e157-42ac-91bd-4b11a5ccbc9f" containerName="main"
Jul 09 17:27:50.990683 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.990531    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="caf5f480-dbf1-4521-b3ba-9f31736c723b" containerName="main"
Jul 09 17:27:50.994693 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.994666    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:50.998747 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.998723    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"default-dockercfg-9gd5z\""
Jul 09 17:27:50.998877 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.998750    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"sub-test-model-ba9b25-kserve-self-signed-certs\""
Jul 09 17:27:50.998877 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.998790    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"openshift-service-ca.crt\""
Jul 09 17:27:50.998877 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:50.998733    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"kube-root-ca.crt\""
Jul 09 17:27:51.004643 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.004619    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"]
Jul 09 17:27:51.030937 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.030912    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:27:51.159226 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.159185    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tmp-dir\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.159226 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.159236    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-dshm\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.159454 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.159341    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvs4z\" (UniqueName: \"kubernetes.io/projected/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-kube-api-access-jvs4z\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.159454 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.159373    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-model-cache\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.159454 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.159397    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-home\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.159454 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.159448    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tls-certs\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260358 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260270    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-home\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260358 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260315    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tls-certs\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260358 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260355    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tmp-dir\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260585 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260390    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-dshm\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260585 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260460    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvs4z\" (UniqueName: \"kubernetes.io/projected/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-kube-api-access-jvs4z\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260585 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260489    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-model-cache\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260807 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260786    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-home\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260867 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260838    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tmp-dir\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.260867 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.260849    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-model-cache\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.262698 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.262675    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-dshm\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.263000 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.262981    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tls-certs\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.268122 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.268099    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvs4z\" (UniqueName: \"kubernetes.io/projected/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-kube-api-access-jvs4z\") pod \"sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") " pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.304293 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.304263    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:27:51.429702 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:51.429675    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"]
Jul 09 17:27:51.431672 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:27:51.431619    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd518e936_d57d_4ebd_8b33_8a6a62e7f0ea.slice/crio-3f37641132dd70b2f655384e9f3e1517aa73ff853be0dd5986af381853f4ab88 WatchSource:0}: Error finding container 3f37641132dd70b2f655384e9f3e1517aa73ff853be0dd5986af381853f4ab88: Status 404 returned error can't find the container with id 3f37641132dd70b2f655384e9f3e1517aa73ff853be0dd5986af381853f4ab88
Jul 09 17:27:52.273554 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:52.273521    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z" event={"ID":"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea","Type":"ContainerStarted","Data":"b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019"}
Jul 09 17:27:52.273932 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:52.273565    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z" event={"ID":"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea","Type":"ContainerStarted","Data":"3f37641132dd70b2f655384e9f3e1517aa73ff853be0dd5986af381853f4ab88"}
Jul 09 17:27:52.291345 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:27:52.291300    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z" podStartSLOduration=2.29128616 podStartE2EDuration="2.29128616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:27:52.290113152 +0000 UTC m=+2219.370054120" watchObservedRunningTime="2026-07-09 17:27:52.29128616 +0000 UTC m=+2219.371227109"
Jul 09 17:28:01.304781 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:01.304741    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:28:01.305257 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:01.304996    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:28:01.318381 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:01.318353    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:28:02.328063 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:02.328032    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:28:29.161974 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:29.161889    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:28:37.741610 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:37.741577    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:28:38.683257 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:38.683223    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"]
Jul 09 17:28:38.704599 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:38.704570    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:28:45.595892 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.595860    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"]
Jul 09 17:28:45.599210 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.599187    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.602959 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.602938    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"rate-test-model-553a9f-kserve-self-signed-certs\""
Jul 09 17:28:45.602959 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.602949    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"openshift-service-ca.crt\""
Jul 09 17:28:45.603127 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.602984    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"kube-root-ca.crt\""
Jul 09 17:28:45.603214 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.603195    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-553a9f\"/\"default-dockercfg-46p5h\""
Jul 09 17:28:45.607430 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.607412    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"]
Jul 09 17:28:45.626886 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.626863    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:28:45.726532 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.726500    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-dshm\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.726532 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.726535    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-tmp-dir\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.726752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.726564    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-model-cache\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.726752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.726632    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-home\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.726752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.726703    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095e3fde-dab7-4b93-8407-7706fca08364-tls-certs\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.726752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.726734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knmrp\" (UniqueName: \"kubernetes.io/projected/095e3fde-dab7-4b93-8407-7706fca08364-kube-api-access-knmrp\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.827886 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.827851    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-dshm\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.827886 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.827890    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-tmp-dir\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.828072 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.827932    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-model-cache\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.828072 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.827964    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-home\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.828137 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.828108    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095e3fde-dab7-4b93-8407-7706fca08364-tls-certs\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.828185 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.828161    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knmrp\" (UniqueName: \"kubernetes.io/projected/095e3fde-dab7-4b93-8407-7706fca08364-kube-api-access-knmrp\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.828437 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.828408    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-model-cache\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.828520 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.828417    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-home\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.828520 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.828470    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-tmp-dir\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.830175 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.830158    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-dshm\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.830507 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.830492    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095e3fde-dab7-4b93-8407-7706fca08364-tls-certs\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.837844 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.837737    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-knmrp\" (UniqueName: \"kubernetes.io/projected/095e3fde-dab7-4b93-8407-7706fca08364-kube-api-access-knmrp\") pod \"rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") " pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:45.908869 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:45.908845    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:46.034949 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:46.034919    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"]
Jul 09 17:28:46.037046 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:28:46.037017    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod095e3fde_dab7_4b93_8407_7706fca08364.slice/crio-d1e4dcc8056d805fe1f229da0dac5ecb5245e85af5f669396a6470bedc555e3b WatchSource:0}: Error finding container d1e4dcc8056d805fe1f229da0dac5ecb5245e85af5f669396a6470bedc555e3b: Status 404 returned error can't find the container with id d1e4dcc8056d805fe1f229da0dac5ecb5245e85af5f669396a6470bedc555e3b
Jul 09 17:28:46.500377 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:46.500345    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k" event={"ID":"095e3fde-dab7-4b93-8407-7706fca08364","Type":"ContainerStarted","Data":"a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42"}
Jul 09 17:28:46.500531 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:46.500389    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k" event={"ID":"095e3fde-dab7-4b93-8407-7706fca08364","Type":"ContainerStarted","Data":"d1e4dcc8056d805fe1f229da0dac5ecb5245e85af5f669396a6470bedc555e3b"}
Jul 09 17:28:46.516776 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:46.516729    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k" podStartSLOduration=1.516714163 podStartE2EDuration="1.516714163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:28:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:28:46.516194286 +0000 UTC m=+2273.596135255" watchObservedRunningTime="2026-07-09 17:28:46.516714163 +0000 UTC m=+2273.596655178"
Jul 09 17:28:53.703086 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.703020    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z" podUID="d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" containerName="main" containerID="cri-o://b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019" gracePeriod=45
Jul 09 17:28:53.966374 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.966319    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:28:53.991703 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.991674    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-model-cache\") pod \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") "
Jul 09 17:28:53.991835 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.991744    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tls-certs\") pod \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") "
Jul 09 17:28:53.991835 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.991775    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-dshm\") pod \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") "
Jul 09 17:28:53.991835 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.991815    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jvs4z\" (UniqueName: \"kubernetes.io/projected/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-kube-api-access-jvs4z\") pod \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") "
Jul 09 17:28:53.992070 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.991857    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tmp-dir\") pod \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") "
Jul 09 17:28:53.992070 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.991885    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-home\") pod \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\" (UID: \"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea\") "
Jul 09 17:28:53.992070 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.991946    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-model-cache" (OuterVolumeSpecName: "model-cache") pod "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" (UID: "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:53.992231 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.992166    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:28:53.992379 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.992345    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" (UID: "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:53.992447 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.992374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-home" (OuterVolumeSpecName: "home") pod "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" (UID: "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:53.994439 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.994411    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-kube-api-access-jvs4z" (OuterVolumeSpecName: "kube-api-access-jvs4z") pod "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" (UID: "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea"). InnerVolumeSpecName "kube-api-access-jvs4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:28:53.994562 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.994440    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" (UID: "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:28:53.994562 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:53.994440    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-dshm" (OuterVolumeSpecName: "dshm") pod "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" (UID: "d518e936-d57d-4ebd-8b33-8a6a62e7f0ea"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:28:54.092559 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.092531    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:28:54.092559 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.092556    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:28:54.092559 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.092565    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvs4z\" (UniqueName: \"kubernetes.io/projected/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-kube-api-access-jvs4z\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:28:54.092823 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.092574    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:28:54.092823 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.092582    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:28:54.514389 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.514340    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:28:54.533735 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.533707    2563 generic.go:356] "Generic (PLEG): container finished" podID="d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" containerID="b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019" exitCode=0
Jul 09 17:28:54.533866 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.533748    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"
Jul 09 17:28:54.533866 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.533792    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z" event={"ID":"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea","Type":"ContainerDied","Data":"b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019"}
Jul 09 17:28:54.533866 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.533833    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z" event={"ID":"d518e936-d57d-4ebd-8b33-8a6a62e7f0ea","Type":"ContainerDied","Data":"3f37641132dd70b2f655384e9f3e1517aa73ff853be0dd5986af381853f4ab88"}
Jul 09 17:28:54.533866 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.533850    2563 scope.go:117] "RemoveContainer" containerID="b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019"
Jul 09 17:28:54.544970 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.544953    2563 scope.go:117] "RemoveContainer" containerID="b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019"
Jul 09 17:28:54.545219 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:28:54.545201    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019\": container with ID starting with b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019 not found: ID does not exist" containerID="b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019"
Jul 09 17:28:54.545267 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.545227    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019"} err="failed to get container status \"b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019\": rpc error: code = NotFound desc = could not find container \"b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019\": container with ID starting with b5146c2b98e04aad5c26a7a24d1b9c22ac92738a81a2135b9b2750440a1e7019 not found: ID does not exist"
Jul 09 17:28:54.556017 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.555997    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"]
Jul 09 17:28:54.561574 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.561551    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/sub-test-model-ba9b25-kserve-7b9dbb4bbf-5pt5z"]
Jul 09 17:28:54.564455 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:54.564434    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:28:55.353082 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:55.353048    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" path="/var/lib/kubelet/pods/d518e936-d57d-4ebd-8b33-8a6a62e7f0ea/volumes"
Jul 09 17:28:55.909549 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:55.909512    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:55.909755 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:55.909563    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:55.922144 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:55.922119    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:28:56.555034 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:28:56.555004    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:29:12.461057 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:12.460992    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:29:12.562198 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:12.562165    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:29:19.899458 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.899417    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"]
Jul 09 17:29:19.899854 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.899711    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" containerName="main"
Jul 09 17:29:19.899854 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.899722    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" containerName="main"
Jul 09 17:29:19.899854 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.899786    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="d518e936-d57d-4ebd-8b33-8a6a62e7f0ea" containerName="main"
Jul 09 17:29:19.902792 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.902778    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:19.907696 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.907670    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"rate-test-model-ba9b25-kserve-self-signed-certs\""
Jul 09 17:29:19.907696 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.907695    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"default-dockercfg-9gd5z\""
Jul 09 17:29:19.907924 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.907734    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"openshift-service-ca.crt\""
Jul 09 17:29:19.907924 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.907750    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-ba9b25\"/\"kube-root-ca.crt\""
Jul 09 17:29:19.914019 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:19.913999    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"]
Jul 09 17:29:20.001049 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.001015    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-model-cache\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.001241 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.001064    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7wv6\" (UniqueName: \"kubernetes.io/projected/ad519770-b76d-4482-92c4-0655e734e93c-kube-api-access-t7wv6\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.001241 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.001188    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-dshm\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.001364 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.001248    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-tmp-dir\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.001364 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.001274    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad519770-b76d-4482-92c4-0655e734e93c-tls-certs\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.001364 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.001318    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-home\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.063502 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.063467    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:29:20.102613 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.102576    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-dshm\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.102826 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.102634    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-tmp-dir\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.102826 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.102678    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad519770-b76d-4482-92c4-0655e734e93c-tls-certs\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.102826 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.102729    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-home\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.102826 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.102770    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-model-cache\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.102826 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.102806    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7wv6\" (UniqueName: \"kubernetes.io/projected/ad519770-b76d-4482-92c4-0655e734e93c-kube-api-access-t7wv6\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.103083 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.103059    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-tmp-dir\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.103155 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.103090    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-home\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.103210 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.103147    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-model-cache\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.104890 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.104872    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-dshm\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.105314 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.105294    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad519770-b76d-4482-92c4-0655e734e93c-tls-certs\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.110717 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.110694    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7wv6\" (UniqueName: \"kubernetes.io/projected/ad519770-b76d-4482-92c4-0655e734e93c-kube-api-access-t7wv6\") pod \"rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") " pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.212163 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.212057    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:20.341151 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.341123    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"]
Jul 09 17:29:20.343471 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:29:20.343438    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podad519770_b76d_4482_92c4_0655e734e93c.slice/crio-6d0e4107d674b873314fd6f7fdf51981cb7d5232af66738b408993d332d542fe WatchSource:0}: Error finding container 6d0e4107d674b873314fd6f7fdf51981cb7d5232af66738b408993d332d542fe: Status 404 returned error can't find the container with id 6d0e4107d674b873314fd6f7fdf51981cb7d5232af66738b408993d332d542fe
Jul 09 17:29:20.642578 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.642545    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2" event={"ID":"ad519770-b76d-4482-92c4-0655e734e93c","Type":"ContainerStarted","Data":"40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716"}
Jul 09 17:29:20.642578 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.642585    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2" event={"ID":"ad519770-b76d-4482-92c4-0655e734e93c","Type":"ContainerStarted","Data":"6d0e4107d674b873314fd6f7fdf51981cb7d5232af66738b408993d332d542fe"}
Jul 09 17:29:20.659337 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:20.659267    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2" podStartSLOduration=1.659250144 podStartE2EDuration="1.659250144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 17:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:29:20.658051703 +0000 UTC m=+2307.737992673" watchObservedRunningTime="2026-07-09 17:29:20.659250144 +0000 UTC m=+2307.739191112"
Jul 09 17:29:30.213176 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:30.213141    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:30.213498 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:30.213225    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:30.225552 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:30.225527    2563 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:30.696513 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:30.696485    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:29:36.774125 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:36.774093    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:29:36.875853 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:36.875814    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:29:45.670449 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:45.670368    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:29:45.771402 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:29:45.771365    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:10.288363 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:10.288332    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"]
Jul 09 17:30:10.306314 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:10.306284    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:11.196936 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:11.196898    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"]
Jul 09 17:30:11.237076 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:11.237024    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:25.308303 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.308238    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k" podUID="095e3fde-dab7-4b93-8407-7706fca08364" containerName="main" containerID="cri-o://a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42" gracePeriod=45
Jul 09 17:30:25.549093 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.549070    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:30:25.658984 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.658957    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-dshm\") pod \"095e3fde-dab7-4b93-8407-7706fca08364\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") "
Jul 09 17:30:25.659194 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659023    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-knmrp\" (UniqueName: \"kubernetes.io/projected/095e3fde-dab7-4b93-8407-7706fca08364-kube-api-access-knmrp\") pod \"095e3fde-dab7-4b93-8407-7706fca08364\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") "
Jul 09 17:30:25.659194 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659076    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-home\") pod \"095e3fde-dab7-4b93-8407-7706fca08364\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") "
Jul 09 17:30:25.659194 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659141    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-model-cache\") pod \"095e3fde-dab7-4b93-8407-7706fca08364\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") "
Jul 09 17:30:25.659194 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659163    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095e3fde-dab7-4b93-8407-7706fca08364-tls-certs\") pod \"095e3fde-dab7-4b93-8407-7706fca08364\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") "
Jul 09 17:30:25.659194 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659192    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-tmp-dir\") pod \"095e3fde-dab7-4b93-8407-7706fca08364\" (UID: \"095e3fde-dab7-4b93-8407-7706fca08364\") "
Jul 09 17:30:25.659613 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659352    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-home" (OuterVolumeSpecName: "home") pod "095e3fde-dab7-4b93-8407-7706fca08364" (UID: "095e3fde-dab7-4b93-8407-7706fca08364"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:25.659613 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659455    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:25.659613 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659437    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-model-cache" (OuterVolumeSpecName: "model-cache") pod "095e3fde-dab7-4b93-8407-7706fca08364" (UID: "095e3fde-dab7-4b93-8407-7706fca08364"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:25.659613 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.659508    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "095e3fde-dab7-4b93-8407-7706fca08364" (UID: "095e3fde-dab7-4b93-8407-7706fca08364"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:25.661176 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.661149    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-dshm" (OuterVolumeSpecName: "dshm") pod "095e3fde-dab7-4b93-8407-7706fca08364" (UID: "095e3fde-dab7-4b93-8407-7706fca08364"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:25.661416 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.661374    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/095e3fde-dab7-4b93-8407-7706fca08364-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "095e3fde-dab7-4b93-8407-7706fca08364" (UID: "095e3fde-dab7-4b93-8407-7706fca08364"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:30:25.661416 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.661393    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/095e3fde-dab7-4b93-8407-7706fca08364-kube-api-access-knmrp" (OuterVolumeSpecName: "kube-api-access-knmrp") pod "095e3fde-dab7-4b93-8407-7706fca08364" (UID: "095e3fde-dab7-4b93-8407-7706fca08364"). InnerVolumeSpecName "kube-api-access-knmrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:25.760045 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.760011    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:25.760045 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.760044    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/095e3fde-dab7-4b93-8407-7706fca08364-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:25.760045 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.760054    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:25.760267 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.760062    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/095e3fde-dab7-4b93-8407-7706fca08364-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:25.760267 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.760071    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knmrp\" (UniqueName: \"kubernetes.io/projected/095e3fde-dab7-4b93-8407-7706fca08364-kube-api-access-knmrp\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:25.923478 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.923390    2563 generic.go:356] "Generic (PLEG): container finished" podID="095e3fde-dab7-4b93-8407-7706fca08364" containerID="a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42" exitCode=0
Jul 09 17:30:25.923478 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.923433    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k" event={"ID":"095e3fde-dab7-4b93-8407-7706fca08364","Type":"ContainerDied","Data":"a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42"}
Jul 09 17:30:25.923478 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.923447    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"
Jul 09 17:30:25.923478 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.923460    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k" event={"ID":"095e3fde-dab7-4b93-8407-7706fca08364","Type":"ContainerDied","Data":"d1e4dcc8056d805fe1f229da0dac5ecb5245e85af5f669396a6470bedc555e3b"}
Jul 09 17:30:25.923478 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.923477    2563 scope.go:117] "RemoveContainer" containerID="a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42"
Jul 09 17:30:25.931719 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.931698    2563 scope.go:117] "RemoveContainer" containerID="a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42"
Jul 09 17:30:25.932000 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:30:25.931982    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42\": container with ID starting with a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42 not found: ID does not exist" containerID="a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42"
Jul 09 17:30:25.932039 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.932009    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42"} err="failed to get container status \"a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42\": rpc error: code = NotFound desc = could not find container \"a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42\": container with ID starting with a1ff6305d152c55ced8a166d0f03aed9e6406760b198c3b5038ad37a5a6e9f42 not found: ID does not exist"
Jul 09 17:30:25.943400 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.943378    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"]
Jul 09 17:30:25.947065 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:25.947045    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-553a9f/rate-test-model-553a9f-kserve-ddc7c5bf9-gz46k"]
Jul 09 17:30:26.216813 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.216722    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2" podUID="ad519770-b76d-4482-92c4-0655e734e93c" containerName="main" containerID="cri-o://40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716" gracePeriod=45
Jul 09 17:30:26.464777 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.464753    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:30:26.567189 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567083    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-model-cache\") pod \"ad519770-b76d-4482-92c4-0655e734e93c\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") "
Jul 09 17:30:26.567189 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567143    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-tmp-dir\") pod \"ad519770-b76d-4482-92c4-0655e734e93c\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") "
Jul 09 17:30:26.567433 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567217    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t7wv6\" (UniqueName: \"kubernetes.io/projected/ad519770-b76d-4482-92c4-0655e734e93c-kube-api-access-t7wv6\") pod \"ad519770-b76d-4482-92c4-0655e734e93c\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") "
Jul 09 17:30:26.567433 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567280    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad519770-b76d-4482-92c4-0655e734e93c-tls-certs\") pod \"ad519770-b76d-4482-92c4-0655e734e93c\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") "
Jul 09 17:30:26.567433 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567324    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-home\") pod \"ad519770-b76d-4482-92c4-0655e734e93c\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") "
Jul 09 17:30:26.567433 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567352    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-dshm\") pod \"ad519770-b76d-4482-92c4-0655e734e93c\" (UID: \"ad519770-b76d-4482-92c4-0655e734e93c\") "
Jul 09 17:30:26.567433 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567413    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-model-cache" (OuterVolumeSpecName: "model-cache") pod "ad519770-b76d-4482-92c4-0655e734e93c" (UID: "ad519770-b76d-4482-92c4-0655e734e93c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:26.567713 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567424    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ad519770-b76d-4482-92c4-0655e734e93c" (UID: "ad519770-b76d-4482-92c4-0655e734e93c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:26.567713 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567604    2563 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-tmp-dir\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:26.567713 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567609    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-home" (OuterVolumeSpecName: "home") pod "ad519770-b76d-4482-92c4-0655e734e93c" (UID: "ad519770-b76d-4482-92c4-0655e734e93c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:26.567713 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.567619    2563 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-model-cache\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:26.569462 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.569439    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-dshm" (OuterVolumeSpecName: "dshm") pod "ad519770-b76d-4482-92c4-0655e734e93c" (UID: "ad519770-b76d-4482-92c4-0655e734e93c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 17:30:26.569553 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.569514    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad519770-b76d-4482-92c4-0655e734e93c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ad519770-b76d-4482-92c4-0655e734e93c" (UID: "ad519770-b76d-4482-92c4-0655e734e93c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:30:26.569614 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.569595    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad519770-b76d-4482-92c4-0655e734e93c-kube-api-access-t7wv6" (OuterVolumeSpecName: "kube-api-access-t7wv6") pod "ad519770-b76d-4482-92c4-0655e734e93c" (UID: "ad519770-b76d-4482-92c4-0655e734e93c"). InnerVolumeSpecName "kube-api-access-t7wv6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:26.668442 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.668394    2563 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad519770-b76d-4482-92c4-0655e734e93c-tls-certs\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:26.668442 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.668437    2563 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-home\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:26.668442 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.668447    2563 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad519770-b76d-4482-92c4-0655e734e93c-dshm\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:26.668442 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.668456    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7wv6\" (UniqueName: \"kubernetes.io/projected/ad519770-b76d-4482-92c4-0655e734e93c-kube-api-access-t7wv6\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:26.929527 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.929492    2563 generic.go:356] "Generic (PLEG): container finished" podID="ad519770-b76d-4482-92c4-0655e734e93c" containerID="40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716" exitCode=0
Jul 09 17:30:26.929752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.929536    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"
Jul 09 17:30:26.929752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.929557    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2" event={"ID":"ad519770-b76d-4482-92c4-0655e734e93c","Type":"ContainerDied","Data":"40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716"}
Jul 09 17:30:26.929752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.929597    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2" event={"ID":"ad519770-b76d-4482-92c4-0655e734e93c","Type":"ContainerDied","Data":"6d0e4107d674b873314fd6f7fdf51981cb7d5232af66738b408993d332d542fe"}
Jul 09 17:30:26.929752 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.929619    2563 scope.go:117] "RemoveContainer" containerID="40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716"
Jul 09 17:30:26.938010 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.937992    2563 scope.go:117] "RemoveContainer" containerID="40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716"
Jul 09 17:30:26.938275 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:30:26.938260    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716\": container with ID starting with 40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716 not found: ID does not exist" containerID="40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716"
Jul 09 17:30:26.938324 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.938279    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716"} err="failed to get container status \"40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716\": rpc error: code = NotFound desc = could not find container \"40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716\": container with ID starting with 40267f38ba9f6c3561ed6cee6548b63714c36fa0591e7b381aad28f62d5fe716 not found: ID does not exist"
Jul 09 17:30:26.949957 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.949934    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"]
Jul 09 17:30:26.953000 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:26.952981    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-ba9b25/rate-test-model-ba9b25-kserve-5b8b6448f5-kpnh2"]
Jul 09 17:30:27.353288 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.353201    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="095e3fde-dab7-4b93-8407-7706fca08364" path="/var/lib/kubelet/pods/095e3fde-dab7-4b93-8407-7706fca08364/volumes"
Jul 09 17:30:27.353587 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.353575    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad519770-b76d-4482-92c4-0655e734e93c" path="/var/lib/kubelet/pods/ad519770-b76d-4482-92c4-0655e734e93c/volumes"
Jul 09 17:30:27.863620 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.863584    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7162-140385809312624"]
Jul 09 17:30:27.864146 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.864026    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="095e3fde-dab7-4b93-8407-7706fca08364" containerName="main"
Jul 09 17:30:27.864146 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.864043    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="095e3fde-dab7-4b93-8407-7706fca08364" containerName="main"
Jul 09 17:30:27.864146 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.864070    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad519770-b76d-4482-92c4-0655e734e93c" containerName="main"
Jul 09 17:30:27.864146 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.864078    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad519770-b76d-4482-92c4-0655e734e93c" containerName="main"
Jul 09 17:30:27.864146 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.864147    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad519770-b76d-4482-92c4-0655e734e93c" containerName="main"
Jul 09 17:30:27.864398 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.864158    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="095e3fde-dab7-4b93-8407-7706fca08364" containerName="main"
Jul 09 17:30:27.868308 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.868286    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7162-140385809312624"
Jul 09 17:30:27.871268 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.871242    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-7vdsn\""
Jul 09 17:30:27.872784 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.872766    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7162-140385809312624"]
Jul 09 17:30:27.979915 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:27.979881    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xbtlj\" (UniqueName: \"kubernetes.io/projected/3c29bf14-8b2b-4ee7-949e-13601d46c769-kube-api-access-xbtlj\") pod \"test-curl-7162-140385809312624\" (UID: \"3c29bf14-8b2b-4ee7-949e-13601d46c769\") " pod="openshift-ingress/test-curl-7162-140385809312624"
Jul 09 17:30:28.080938 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:28.080903    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xbtlj\" (UniqueName: \"kubernetes.io/projected/3c29bf14-8b2b-4ee7-949e-13601d46c769-kube-api-access-xbtlj\") pod \"test-curl-7162-140385809312624\" (UID: \"3c29bf14-8b2b-4ee7-949e-13601d46c769\") " pod="openshift-ingress/test-curl-7162-140385809312624"
Jul 09 17:30:28.089387 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:28.089354    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xbtlj\" (UniqueName: \"kubernetes.io/projected/3c29bf14-8b2b-4ee7-949e-13601d46c769-kube-api-access-xbtlj\") pod \"test-curl-7162-140385809312624\" (UID: \"3c29bf14-8b2b-4ee7-949e-13601d46c769\") " pod="openshift-ingress/test-curl-7162-140385809312624"
Jul 09 17:30:28.178338 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:28.178244    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7162-140385809312624"
Jul 09 17:30:28.301672 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:28.301629    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7162-140385809312624"]
Jul 09 17:30:28.303851 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:30:28.303823    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c29bf14_8b2b_4ee7_949e_13601d46c769.slice/crio-57d174e25dbebac57972ec3c009f4ec9f3ce88e996036248c9cc49f25d942960 WatchSource:0}: Error finding container 57d174e25dbebac57972ec3c009f4ec9f3ce88e996036248c9cc49f25d942960: Status 404 returned error can't find the container with id 57d174e25dbebac57972ec3c009f4ec9f3ce88e996036248c9cc49f25d942960
Jul 09 17:30:28.305517 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:28.305497    2563 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 17:30:28.939837 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:28.939801    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7162-140385809312624" event={"ID":"3c29bf14-8b2b-4ee7-949e-13601d46c769","Type":"ContainerStarted","Data":"57d174e25dbebac57972ec3c009f4ec9f3ce88e996036248c9cc49f25d942960"}
Jul 09 17:30:29.946010 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:29.945975    2563 generic.go:356] "Generic (PLEG): container finished" podID="3c29bf14-8b2b-4ee7-949e-13601d46c769" containerID="8a8e3ff2c71befb93a6dc7a22d4bf3e3e6e9d4bd3560cb63b6abe2ea45362732" exitCode=0
Jul 09 17:30:29.946371 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:29.946039    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7162-140385809312624" event={"ID":"3c29bf14-8b2b-4ee7-949e-13601d46c769","Type":"ContainerDied","Data":"8a8e3ff2c71befb93a6dc7a22d4bf3e3e6e9d4bd3560cb63b6abe2ea45362732"}
Jul 09 17:30:31.083545 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.083522    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7162-140385809312624"
Jul 09 17:30:31.105818 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.105788    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7162-140385809312624_3c29bf14-8b2b-4ee7-949e-13601d46c769/test-curl-7162-140385809312624/0.log"
Jul 09 17:30:31.126460 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.126429    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7162-140385809312624"]
Jul 09 17:30:31.128853 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.128826    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7162-140385809312624"]
Jul 09 17:30:31.205576 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.205487    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xbtlj\" (UniqueName: \"kubernetes.io/projected/3c29bf14-8b2b-4ee7-949e-13601d46c769-kube-api-access-xbtlj\") pod \"3c29bf14-8b2b-4ee7-949e-13601d46c769\" (UID: \"3c29bf14-8b2b-4ee7-949e-13601d46c769\") "
Jul 09 17:30:31.207521 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.207497    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c29bf14-8b2b-4ee7-949e-13601d46c769-kube-api-access-xbtlj" (OuterVolumeSpecName: "kube-api-access-xbtlj") pod "3c29bf14-8b2b-4ee7-949e-13601d46c769" (UID: "3c29bf14-8b2b-4ee7-949e-13601d46c769"). InnerVolumeSpecName "kube-api-access-xbtlj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:31.306941 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.306909    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xbtlj\" (UniqueName: \"kubernetes.io/projected/3c29bf14-8b2b-4ee7-949e-13601d46c769-kube-api-access-xbtlj\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:31.352665 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.352614    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c29bf14-8b2b-4ee7-949e-13601d46c769" path="/var/lib/kubelet/pods/3c29bf14-8b2b-4ee7-949e-13601d46c769/volumes"
Jul 09 17:30:31.353511 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.353487    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:31.754391 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.754357    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:31.854761 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.854725    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:31.957462 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.957436    2563 scope.go:117] "RemoveContainer" containerID="8a8e3ff2c71befb93a6dc7a22d4bf3e3e6e9d4bd3560cb63b6abe2ea45362732"
Jul 09 17:30:31.957462 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:31.957461    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7162-140385809312624"
Jul 09 17:30:32.027393 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:32.027296    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:37.929787 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:37.929747    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:38.030008 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:38.029964    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:38.204054 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:38.203950    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:38.477612 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:38.477502    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:38.601808 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:38.601781    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:39.000581 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.000551    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:39.100696 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.100633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:39.202168 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.202128    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:39.266590 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.266426    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"]
Jul 09 17:30:39.266770 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.266697    2563 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" podUID="5b9e7f5d-3bc9-47a4-9764-275c63091356" containerName="maas-api" containerID="cri-o://c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce" gracePeriod=30
Jul 09 17:30:39.312709 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.312633    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:39.715146 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.715120    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:30:39.882338 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.882291    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls\") pod \"5b9e7f5d-3bc9-47a4-9764-275c63091356\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") "
Jul 09 17:30:39.882519 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.882357    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e7f5d-3bc9-47a4-9764-275c63091356-openshift-service-ca\") pod \"5b9e7f5d-3bc9-47a4-9764-275c63091356\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") "
Jul 09 17:30:39.882519 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.882384    2563 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/5b9e7f5d-3bc9-47a4-9764-275c63091356-kube-api-access-6wsgr\") pod \"5b9e7f5d-3bc9-47a4-9764-275c63091356\" (UID: \"5b9e7f5d-3bc9-47a4-9764-275c63091356\") "
Jul 09 17:30:39.882852 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.882831    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b9e7f5d-3bc9-47a4-9764-275c63091356-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5b9e7f5d-3bc9-47a4-9764-275c63091356" (UID: "5b9e7f5d-3bc9-47a4-9764-275c63091356"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 17:30:39.884572 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.884546    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5b9e7f5d-3bc9-47a4-9764-275c63091356" (UID: "5b9e7f5d-3bc9-47a4-9764-275c63091356"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 17:30:39.884644 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.884572    2563 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b9e7f5d-3bc9-47a4-9764-275c63091356-kube-api-access-6wsgr" (OuterVolumeSpecName: "kube-api-access-6wsgr") pod "5b9e7f5d-3bc9-47a4-9764-275c63091356" (UID: "5b9e7f5d-3bc9-47a4-9764-275c63091356"). InnerVolumeSpecName "kube-api-access-6wsgr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 17:30:39.984166 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.984133    2563 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5b9e7f5d-3bc9-47a4-9764-275c63091356-maas-api-tls\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:39.984166 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.984159    2563 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5b9e7f5d-3bc9-47a4-9764-275c63091356-openshift-service-ca\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:39.984166 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.984169    2563 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6wsgr\" (UniqueName: \"kubernetes.io/projected/5b9e7f5d-3bc9-47a4-9764-275c63091356-kube-api-access-6wsgr\") on node \"ip-10-0-139-29.ec2.internal\" DevicePath \"\""
Jul 09 17:30:39.993779 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.993752    2563 generic.go:356] "Generic (PLEG): container finished" podID="5b9e7f5d-3bc9-47a4-9764-275c63091356" containerID="c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce" exitCode=0
Jul 09 17:30:39.993931 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.993800    2563 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"
Jul 09 17:30:39.993931 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.993837    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" event={"ID":"5b9e7f5d-3bc9-47a4-9764-275c63091356","Type":"ContainerDied","Data":"c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce"}
Jul 09 17:30:39.993931 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.993889    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx" event={"ID":"5b9e7f5d-3bc9-47a4-9764-275c63091356","Type":"ContainerDied","Data":"370414ef1be4f320f8272f22be99a972cb9fb0c29cb47340e4e65e2e256c428c"}
Jul 09 17:30:39.993931 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:39.993911    2563 scope.go:117] "RemoveContainer" containerID="c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce"
Jul 09 17:30:40.002518 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:40.002492    2563 scope.go:117] "RemoveContainer" containerID="c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce"
Jul 09 17:30:40.002875 ip-10-0-139-29 kubenswrapper[2563]: E0709 17:30:40.002845    2563 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce\": container with ID starting with c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce not found: ID does not exist" containerID="c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce"
Jul 09 17:30:40.002917 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:40.002869    2563 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce"} err="failed to get container status \"c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce\": rpc error: code = NotFound desc = could not find container \"c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce\": container with ID starting with c4a9de578f3e35fee9ef559d8295ae4090fca3b3ff96a741ccee40f585bcffce not found: ID does not exist"
Jul 09 17:30:40.018379 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:40.018358    2563 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"]
Jul 09 17:30:40.022351 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:40.022331    2563 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-ba9b25-5bc78c8c95-hvcgx"]
Jul 09 17:30:41.352572 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:41.352537    2563 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b9e7f5d-3bc9-47a4-9764-275c63091356" path="/var/lib/kubelet/pods/5b9e7f5d-3bc9-47a4-9764-275c63091356/volumes"
Jul 09 17:30:44.925107 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:44.925072    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:45.011557 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:45.011525    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:45.704279 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:45.704250    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:45.795504 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:45.795471    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:45.824906 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:45.824873    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfb7d9596-dwlht_fb800722-ecc1-421e-8627-d7d086f2c9af/authorino/0.log"
Jul 09 17:30:50.996010 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:50.995970    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8lnbc_06bb29c5-4dfd-42cc-944a-563d29c04fc5/manager/0.log"
Jul 09 17:30:51.104617 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:51.104590    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fdb769df8-6wzmg_b4ff4442-d40e-418f-8bf7-f3da82940523/copy-manifests/0.log"
Jul 09 17:30:51.116839 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:51.116815    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fdb769df8-6wzmg_b4ff4442-d40e-418f-8bf7-f3da82940523/manager/0.log"
Jul 09 17:30:53.375634 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:53.375604    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfb7d9596-dwlht_fb800722-ecc1-421e-8627-d7d086f2c9af/authorino/0.log"
Jul 09 17:30:53.721746 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:53.721638    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bpjgk_48a6abca-cf18-4339-9831-e5e7217c34fb/kuadrant-console-plugin/0.log"
Jul 09 17:30:54.898147 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:54.898119    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b94bd4c99-6lpfl_0c721106-f6c3-446e-bd65-82f745348116/kube-auth-proxy/0.log"
Jul 09 17:30:55.697069 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:55.697034    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-67d8d5d6d-llcf7_5d594c3e-1304-431c-8b24-84ba41017b74/main/0.log"
Jul 09 17:30:56.134942 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:56.134914    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-vcwpt_e6958f86-fd5e-4c6a-9ca5-37782a3a303c/main/0.log"
Jul 09 17:30:56.243754 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:56.243723    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-5fc6fb456f-5s9rm_a1c94d5d-7b3a-4046-91d7-e75a8d046474/main/0.log"
Jul 09 17:30:59.760428 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:59.760387    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:30:59.827517 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:30:59.827476    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:31:02.527388 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:02.527357    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-cf425_ae1eeec8-a184-4829-b055-c7715ca287ff/global-pull-secret-syncer/0.log"
Jul 09 17:31:02.661872 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:02.661837    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log"
Jul 09 17:31:02.787207 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:02.787128    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-139-29.ec2.internal_fc2d5eb401e4b3f52c292a48eec32885/haproxy/0.log"
Jul 09 17:31:07.954576 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:07.954544    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-5dfb7d9596-dwlht_fb800722-ecc1-421e-8627-d7d086f2c9af/authorino/0.log"
Jul 09 17:31:08.046533 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:08.046504    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-bpjgk_48a6abca-cf18-4339-9831-e5e7217c34fb/kuadrant-console-plugin/0.log"
Jul 09 17:31:09.856800 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:09.856765    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:31:10.595285 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:10.595254    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ch47v_270a2f78-b8fa-4ea2-bba9-fa3b9fed0741/node-exporter/0.log"
Jul 09 17:31:10.615625 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:10.615599    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ch47v_270a2f78-b8fa-4ea2-bba9-fa3b9fed0741/kube-rbac-proxy/0.log"
Jul 09 17:31:10.637599 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:10.637575    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-ch47v_270a2f78-b8fa-4ea2-bba9-fa3b9fed0741/init-textfile/0.log"
Jul 09 17:31:12.363567 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.363525    2563 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"]
Jul 09 17:31:12.364079 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.363966    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b9e7f5d-3bc9-47a4-9764-275c63091356" containerName="maas-api"
Jul 09 17:31:12.364079 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.363984    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b9e7f5d-3bc9-47a4-9764-275c63091356" containerName="maas-api"
Jul 09 17:31:12.364079 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.364005    2563 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c29bf14-8b2b-4ee7-949e-13601d46c769" containerName="test-curl-7162-140385809312624"
Jul 09 17:31:12.364079 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.364014    2563 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c29bf14-8b2b-4ee7-949e-13601d46c769" containerName="test-curl-7162-140385809312624"
Jul 09 17:31:12.364295 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.364090    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b9e7f5d-3bc9-47a4-9764-275c63091356" containerName="maas-api"
Jul 09 17:31:12.364295 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.364101    2563 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c29bf14-8b2b-4ee7-949e-13601d46c769" containerName="test-curl-7162-140385809312624"
Jul 09 17:31:12.369302 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.369282    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.374416 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.374394    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7jhs\"/\"kube-root-ca.crt\""
Jul 09 17:31:12.375827 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.375507    2563 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-l7jhs\"/\"openshift-service-ca.crt\""
Jul 09 17:31:12.375827 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.375507    2563 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-l7jhs\"/\"default-dockercfg-nkppl\""
Jul 09 17:31:12.379969 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.379949    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"]
Jul 09 17:31:12.433770 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.433734    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-podres\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.433993 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.433791    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-proc\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.433993 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.433821    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-sys\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.433993 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.433854    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-lib-modules\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.433993 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.433879    2563 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg9dx\" (UniqueName: \"kubernetes.io/projected/953ccc28-60ea-4d7e-a949-24d4207233b1-kube-api-access-fg9dx\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535083 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535043    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-proc\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535240 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535087    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-proc\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535240 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535096    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-sys\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535240 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535138    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-lib-modules\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535240 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535142    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-sys\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535240 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535171    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg9dx\" (UniqueName: \"kubernetes.io/projected/953ccc28-60ea-4d7e-a949-24d4207233b1-kube-api-access-fg9dx\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535240 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535233    2563 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-podres\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535442 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535290    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-lib-modules\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.535442 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.535366    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/953ccc28-60ea-4d7e-a949-24d4207233b1-podres\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.546216 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.546191    2563 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg9dx\" (UniqueName: \"kubernetes.io/projected/953ccc28-60ea-4d7e-a949-24d4207233b1-kube-api-access-fg9dx\") pod \"perf-node-gather-daemonset-9522g\" (UID: \"953ccc28-60ea-4d7e-a949-24d4207233b1\") " pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.680373 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.680270    2563 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:12.820926 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:12.820864    2563 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"]
Jul 09 17:31:12.823542 ip-10-0-139-29 kubenswrapper[2563]: W0709 17:31:12.823513    2563 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod953ccc28_60ea_4d7e_a949_24d4207233b1.slice/crio-6ce59ba84475ce7b11dd447d08d19f7e481c988a6a8475aed0f10ee75134a46e WatchSource:0}: Error finding container 6ce59ba84475ce7b11dd447d08d19f7e481c988a6a8475aed0f10ee75134a46e: Status 404 returned error can't find the container with id 6ce59ba84475ce7b11dd447d08d19f7e481c988a6a8475aed0f10ee75134a46e
Jul 09 17:31:13.127734 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:13.127696    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g" event={"ID":"953ccc28-60ea-4d7e-a949-24d4207233b1","Type":"ContainerStarted","Data":"792e80d3b129fdeebe3de5f695710e863334ebef1d83ea2984929d6acf58dfe2"}
Jul 09 17:31:13.127970 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:13.127749    2563 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g" event={"ID":"953ccc28-60ea-4d7e-a949-24d4207233b1","Type":"ContainerStarted","Data":"6ce59ba84475ce7b11dd447d08d19f7e481c988a6a8475aed0f10ee75134a46e"}
Jul 09 17:31:13.127970 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:13.127778    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:13.146683 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:13.146619    2563 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g" podStartSLOduration=1.146601517 podStartE2EDuration="1.146601517s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 17:31:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 17:31:13.145039513 +0000 UTC m=+2420.224980482" watchObservedRunningTime="2026-07-09 17:31:13.146601517 +0000 UTC m=+2420.226542486"
Jul 09 17:31:15.573072 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:15.572997    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wgwn9_7f9225d8-e78f-4897-aeb8-d382e7f9b718/dns/0.log"
Jul 09 17:31:15.592915 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:15.592895    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-wgwn9_7f9225d8-e78f-4897-aeb8-d382e7f9b718/kube-rbac-proxy/0.log"
Jul 09 17:31:15.660018 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:15.659990    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-qkg46_39cd66d1-cdef-4ed1-b094-5d664f273801/dns-node-resolver/0.log"
Jul 09 17:31:16.186237 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:16.186207    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-7f5469cd45-w4zzd_49287bb3-300f-43f0-ac93-00fdf9a2fa99/registry/0.log"
Jul 09 17:31:16.245058 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:16.245031    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-tgdbb_d9a3dd23-a51f-42b8-b4d5-fb95eb681add/node-ca/0.log"
Jul 09 17:31:17.422347 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:17.422321    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b94bd4c99-6lpfl_0c721106-f6c3-446e-bd65-82f745348116/kube-auth-proxy/0.log"
Jul 09 17:31:18.372479 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:18.372453    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cpsnp_055cde3f-5066-4497-8676-304897be9ff2/serve-healthcheck-canary/0.log"
Jul 09 17:31:18.956511 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:18.956486    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9mbt_a1b9d36e-47f6-43cf-af37-c3ce7f486e34/kube-rbac-proxy/0.log"
Jul 09 17:31:18.975512 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:18.975481    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9mbt_a1b9d36e-47f6-43cf-af37-c3ce7f486e34/exporter/0.log"
Jul 09 17:31:18.994500 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:18.994473    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-m9mbt_a1b9d36e-47f6-43cf-af37-c3ce7f486e34/extractor/0.log"
Jul 09 17:31:19.140506 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:19.140481    2563 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-l7jhs/perf-node-gather-daemonset-9522g"
Jul 09 17:31:20.945110 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:20.945078    2563 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6ccpn_20ef3d43-c5db-49ef-8141-5cfbd35f397a/konnectivity-agent/0.log" limit=1024
Jul 09 17:31:21.731661 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:21.731624    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-8lnbc_06bb29c5-4dfd-42cc-944a-563d29c04fc5/manager/0.log"
Jul 09 17:31:21.805351 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:21.805328    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fdb769df8-6wzmg_b4ff4442-d40e-418f-8bf7-f3da82940523/manager/0.log"
Jul 09 17:31:21.824743 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:21.824722    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-fdb769df8-6wzmg_b4ff4442-d40e-418f-8bf7-f3da82940523/copy-manifests/0.log"
Jul 09 17:31:23.181442 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:23.181415    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5c969b7c4d-ft6dw_d6a3ad51-e423-4f2f-818b-fece3dd0c742/manager/0.log"
Jul 09 17:31:23.356525 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:23.356498    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b94bd4c99-6lpfl_0c721106-f6c3-446e-bd65-82f745348116/kube-auth-proxy/0.log"
Jul 09 17:31:24.942636 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:24.942596    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b94bd4c99-6lpfl_0c721106-f6c3-446e-bd65-82f745348116/kube-auth-proxy/0.log"
Jul 09 17:31:28.070537 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:28.070504    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-kj7cd_17eafe93-1fc4-426a-85f8-4ee1c9e2136b/migrator/0.log"
Jul 09 17:31:28.089440 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:28.089414    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-kj7cd_17eafe93-1fc4-426a-85f8-4ee1c9e2136b/graceful-termination/0.log"
Jul 09 17:31:29.382746 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:29.382718    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2stnr_628ba919-14e4-4c4b-928c-f3518abff7e1/kube-multus-additional-cni-plugins/0.log"
Jul 09 17:31:29.403139 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:29.403110    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2stnr_628ba919-14e4-4c4b-928c-f3518abff7e1/egress-router-binary-copy/0.log"
Jul 09 17:31:29.422841 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:29.422815    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2stnr_628ba919-14e4-4c4b-928c-f3518abff7e1/cni-plugins/0.log"
Jul 09 17:31:29.441944 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:29.441919    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2stnr_628ba919-14e4-4c4b-928c-f3518abff7e1/bond-cni-plugin/0.log"
Jul 09 17:31:29.461470 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:29.461440    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2stnr_628ba919-14e4-4c4b-928c-f3518abff7e1/routeoverride-cni/0.log"
Jul 09 17:31:29.480120 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:29.480096    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2stnr_628ba919-14e4-4c4b-928c-f3518abff7e1/whereabouts-cni-bincopy/0.log"
Jul 09 17:31:29.499849 ip-10-0-139-29 kubenswrapper[2563]: I0709 17:31:29.499820    2563 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2stnr_628ba919-14e4-4c4b-928c-f3518abff7e1/whereabouts-cni/0.log"