Jul 09 18:53:09.543838 ip-10-0-129-93 systemd[1]: Starting Kubernetes Kubelet...
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.884218    2556 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888333    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888347    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888350    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888353    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888356    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888359    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888362    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:53:09.936004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888365    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888368    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888370    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888373    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888375    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888378    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888380    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888384    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888387    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888389    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888394    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888398    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888401    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888404    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888407    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888410    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888412    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888415    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888417    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888420    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:53:09.937277 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888422    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888425    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888428    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888431    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888434    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888437    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888439    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888442    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888444    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888448    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888450    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888453    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888456    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888458    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888461    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888464    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888466    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888469    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888472    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888474    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:53:09.938155 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888477    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888479    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888482    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888485    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888487    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888491    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888493    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888496    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888499    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888501    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888503    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888506    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888508    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888511    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888514    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888517    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888519    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888522    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888539    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888542    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:53:09.939016 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888544    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888547    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888550    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888554    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888556    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888559    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888563    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888567    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888571    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888575    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888578    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888581    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888583    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888586    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888588    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888591    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888594    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888596    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888599    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:53:09.939880 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888601    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888606    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888609    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888612    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888615    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888617    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888620    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888623    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888626    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888628    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888631    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888634    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.888637    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889363    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889369    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889372    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889374    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889379    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889381    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889384    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:53:09.940757 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889387    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889391    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889393    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889396    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889398    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889401    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889404    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889407    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889409    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889412    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889414    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889417    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889419    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889422    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889424    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889427    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889429    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889432    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889435    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889437    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:53:09.941622 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889440    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889442    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889444    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889447    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889449    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889454    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889456    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889460    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889463    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889465    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889468    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889472    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889475    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889478    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889480    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889483    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889485    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889488    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889492    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:53:09.942495 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889496    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889499    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889502    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889505    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889507    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889510    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889514    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889517    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889521    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889540    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889544    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889547    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889549    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889552    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889555    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889557    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889561    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889563    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889567    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:53:09.943369 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889570    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889572    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889575    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889578    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889580    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889584    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889587    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889590    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889594    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889597    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889599    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889602    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889604    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889607    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889610    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889613    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889616    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889619    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889622    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:53:09.944227 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889624    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889627    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889630    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889632    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889635    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889637    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889640    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889642    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889644    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889647    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889649    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889652    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889655    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889658    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.889660    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890065    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890069    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890072    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890076    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890079    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:53:09.945099 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890083    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890085    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890088    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890091    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890095    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890098    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890101    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890104    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890107    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890110    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890113    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890115    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890118    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890121    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890124    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890126    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890129    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890132    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890134    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:53:09.946164 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890138    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890141    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890144    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890147    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890149    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890153    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890156    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890158    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890161    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890163    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890166    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890169    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890221    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890273    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890276    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890279    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890282    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890285    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890288    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890291    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:53:09.947035 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890294    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890297    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890299    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890302    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890305    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890308    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890314    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890318    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890321    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890323    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890326    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890329    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890332    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.890335    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891046    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891069    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891076    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891082    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891087    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891092    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:53:09.947900 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891098    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891102    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891107    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891112    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891116    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891121    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891132    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891136    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891141    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891145    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891149    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891153    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891158    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891166    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891204    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891215    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891222    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891228    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891233    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:53:09.948782 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891265    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891404    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891412    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891418    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891423    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891426    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891430    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891433    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891436    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891438    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891441    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891444    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891448    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891451    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891454    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.891457    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891572    2556 flags.go:64] FLAG: --address="0.0.0.0"
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891583    2556 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891591    2556 flags.go:64] FLAG: --anonymous-auth="true"
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891595    2556 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891600    2556 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 09 18:53:09.949677 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891603    2556 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891607    2556 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891612    2556 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891615    2556 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891618    2556 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891625    2556 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891629    2556 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891632    2556 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891635    2556 flags.go:64] FLAG: --cgroup-root=""
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891638    2556 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891641    2556 flags.go:64] FLAG: --client-ca-file=""
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891643    2556 flags.go:64] FLAG: --cloud-provider="external"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891646    2556 flags.go:64] FLAG: --cluster-dns="[]"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891652    2556 flags.go:64] FLAG: --cluster-domain=""
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891655    2556 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891658    2556 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891661    2556 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891664    2556 flags.go:64] FLAG: --container-log-max-files="5"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891668    2556 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891671    2556 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891674    2556 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891677    2556 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891680    2556 flags.go:64] FLAG: --contention-profiling="false"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891683    2556 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 09 18:53:09.950553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891686    2556 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891694    2556 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891697    2556 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891701    2556 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891704    2556 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891707    2556 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891710    2556 flags.go:64] FLAG: --enable-load-reader="false"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891713    2556 flags.go:64] FLAG: --enable-server="true"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891716    2556 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891735    2556 flags.go:64] FLAG: --event-burst="100"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891739    2556 flags.go:64] FLAG: --event-qps="50"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891742    2556 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891746    2556 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891750    2556 flags.go:64] FLAG: --eviction-hard=""
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891755    2556 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891758    2556 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891761    2556 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891764    2556 flags.go:64] FLAG: --eviction-soft=""
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891767    2556 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891770    2556 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891773    2556 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891776    2556 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891778    2556 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891781    2556 flags.go:64] FLAG: --fail-swap-on="true"
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891785    2556 flags.go:64] FLAG: --feature-gates=""
Jul 09 18:53:09.952060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891789    2556 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891792    2556 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891795    2556 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891798    2556 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891801    2556 flags.go:64] FLAG: --healthz-port="10248"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891804    2556 flags.go:64] FLAG: --help="false"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891807    2556 flags.go:64] FLAG: --hostname-override="ip-10-0-129-93.ec2.internal"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891810    2556 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891815    2556 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891818    2556 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891821    2556 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891825    2556 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891828    2556 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891830    2556 flags.go:64] FLAG: --image-service-endpoint=""
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891833    2556 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891836    2556 flags.go:64] FLAG: --kube-api-burst="100"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891839    2556 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891842    2556 flags.go:64] FLAG: --kube-api-qps="50"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891845    2556 flags.go:64] FLAG: --kube-reserved=""
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891847    2556 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891851    2556 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891854    2556 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891857    2556 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891860    2556 flags.go:64] FLAG: --lock-file=""
Jul 09 18:53:09.956467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891863    2556 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 09 18:53:09.954582 ip-10-0-129-93 systemd[1]: Started Kubernetes Kubelet.
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891866    2556 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891869    2556 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891874    2556 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891877    2556 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891880    2556 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891882    2556 flags.go:64] FLAG: --logging-format="text"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891885    2556 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891888    2556 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891891    2556 flags.go:64] FLAG: --manifest-url=""
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891894    2556 flags.go:64] FLAG: --manifest-url-header=""
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891898    2556 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891901    2556 flags.go:64] FLAG: --max-open-files="1000000"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891906    2556 flags.go:64] FLAG: --max-pods="110"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891909    2556 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891911    2556 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891916    2556 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891919    2556 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891921    2556 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891924    2556 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891927    2556 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891934    2556 flags.go:64] FLAG: --node-status-max-images="50"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891937    2556 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891941    2556 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 09 18:53:09.957783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891944    2556 flags.go:64] FLAG: --pod-cidr=""
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891947    2556 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891953    2556 flags.go:64] FLAG: --pod-manifest-path=""
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891956    2556 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891959    2556 flags.go:64] FLAG: --pods-per-core="0"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891963    2556 flags.go:64] FLAG: --port="10250"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891967    2556 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891970    2556 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-012ac0c86cb56b75e"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891973    2556 flags.go:64] FLAG: --qos-reserved=""
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891976    2556 flags.go:64] FLAG: --read-only-port="10255"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891979    2556 flags.go:64] FLAG: --register-node="true"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891982    2556 flags.go:64] FLAG: --register-with-taints=""
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891986    2556 flags.go:64] FLAG: --registry-burst="10"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891988    2556 flags.go:64] FLAG: --registry-qps="5"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891991    2556 flags.go:64] FLAG: --reserved-cpus=""
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891994    2556 flags.go:64] FLAG: --reserved-memory=""
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.891998    2556 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892001    2556 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892004    2556 flags.go:64] FLAG: --rotate-certificates="false"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892007    2556 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892010    2556 flags.go:64] FLAG: --runonce="false"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892013    2556 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892016    2556 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892018    2556 flags.go:64] FLAG: --seccomp-default="false"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892023    2556 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 09 18:53:10.176117 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892026    2556 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892029    2556 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892031    2556 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892034    2556 flags.go:64] FLAG: --storage-driver-password="root"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892037    2556 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892040    2556 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892043    2556 flags.go:64] FLAG: --storage-driver-user="root"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892046    2556 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892049    2556 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892052    2556 flags.go:64] FLAG: --system-cgroups=""
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892055    2556 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892061    2556 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892066    2556 flags.go:64] FLAG: --tls-cert-file=""
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892069    2556 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892073    2556 flags.go:64] FLAG: --tls-min-version=""
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892076    2556 flags.go:64] FLAG: --tls-private-key-file=""
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892079    2556 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892081    2556 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892084    2556 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892087    2556 flags.go:64] FLAG: --v="2"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892091    2556 flags.go:64] FLAG: --version="false"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892096    2556 flags.go:64] FLAG: --vmodule=""
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892099    2556 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.892102    2556 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892224    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:53:10.198111 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892227    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892231    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892234    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892236    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892239    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892242    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892248    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892251    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892254    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892257    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892260    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892263    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892265    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892268    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892270    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892273    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892276    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892278    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892281    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892284    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:53:10.199574 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892287    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892289    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892292    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892295    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892298    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892300    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892303    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892305    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892308    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892310    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892313    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892316    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892319    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892322    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892324    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892327    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892329    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892332    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892336    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892338    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:53:10.200742 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892341    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892344    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892346    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892349    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892351    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892354    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892356    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892359    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892363    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892367    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892369    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892375    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892379    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892381    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892384    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892387    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892390    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892394    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892397    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:53:10.201898 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892400    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892403    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892406    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892408    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892411    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892413    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892416    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892420    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892422    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892425    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892427    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892431    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892433    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892436    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892439    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892441    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892443    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892446    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892448    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:53:10.203055 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892451    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892454    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892456    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892459    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892462    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892466    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892469    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892472    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892474    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892477    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892479    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892482    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892484    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892487    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892489    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892492    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892495    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892497    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892499    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:53:10.204210 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.892502    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.893001    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.899962    2556 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.899980    2556 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.900667    2556 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.900675    2556 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900738    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900743    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900747    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900752    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900756    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900759    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900762    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900765    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:53:10.205381 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900768    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900771    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900773    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900776    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900779    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900781    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900785    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900789    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900792    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900796    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900799    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900802    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900805    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900808    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900811    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900814    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900817    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900821    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900824    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:53:10.206312 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900827    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900830    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900833    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900836    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900839    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900842    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900847    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900849    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900852    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900854    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900857    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900860    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900863    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900865    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900868    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900871    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900873    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900876    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900878    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900881    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:53:10.207488 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900883    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900886    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900889    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900892    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900895    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900897    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900900    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900902    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900905    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900908    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900911    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900914    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900916    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900919    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900922    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900926    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900929    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900931    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900934    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900936    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:53:10.208881 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900939    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900941    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900947    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900950    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900953    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900955    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900958    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900968    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900971    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900974    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900977    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900979    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900982    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900985    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900988    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900991    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900994    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900996    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.900999    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:53:10.210065 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901002    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901005    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901008    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901010    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901013    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901016    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901018    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901021    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901023    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901027    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901030    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901032    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901035    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.901040    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901176    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 09 18:53:10.211004 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901183    2556 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901187    2556 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901190    2556 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901193    2556 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901196    2556 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901199    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901201    2556 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901204    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901206    2556 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901209    2556 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901212    2556 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901214    2556 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901217    2556 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901219    2556 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901222    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901225    2556 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901227    2556 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901230    2556 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901232    2556 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 09 18:53:10.211957 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901235    2556 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901240    2556 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901244    2556 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901247    2556 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901250    2556 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901253    2556 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901255    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901258    2556 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901261    2556 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901263    2556 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901266    2556 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901269    2556 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901272    2556 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901276    2556 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901279    2556 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901282    2556 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901284    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901287    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901289    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 09 18:53:10.212888 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901292    2556 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901294    2556 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901297    2556 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901299    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901302    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901305    2556 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901307    2556 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901310    2556 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901312    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901315    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901317    2556 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901320    2556 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901322    2556 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901325    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901330    2556 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901333    2556 feature_gate.go:328] unrecognized feature gate: Example2
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901336    2556 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901339    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901342    2556 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901345    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 09 18:53:10.213849 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901347    2556 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901350    2556 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901353    2556 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901356    2556 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901359    2556 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901362    2556 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901366    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901368    2556 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901371    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901373    2556 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901376    2556 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901378    2556 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901381    2556 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901384    2556 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901386    2556 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901389    2556 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901391    2556 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901394    2556 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901396    2556 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901399    2556 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 09 18:53:10.214804 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901401    2556 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901404    2556 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901406    2556 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901409    2556 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901411    2556 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901413    2556 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901419    2556 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901422    2556 feature_gate.go:328] unrecognized feature gate: Example
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901425    2556 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901428    2556 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901431    2556 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901434    2556 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901436    2556 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901439    2556 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901441    2556 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901444    2556 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901447    2556 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901449    2556 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901453    2556 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 09 18:53:10.215764 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:09.901456    2556 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.901460    2556 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.901611    2556 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.904356    2556 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.905633    2556 server.go:1019] "Starting client certificate rotation"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.905735    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.905795    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.907759    2556 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.909008    2556 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.917965    2556 log.go:25] "Validated CRI v1 runtime API"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.922305    2556 log.go:25] "Validated CRI v1 image API"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.925972    2556 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 09 18:53:10.216928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.928486    2556 fs.go:135] Filesystem UUIDs: map[5fb0d8cb-21e1-4020-b9ef-666d3efc4ada:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 95c4d291-44f0-4ed4-aa39-8c76ed4cb069:/dev/nvme0n1p4]
Jul 09 18:53:10.217637 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.928503    2556 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.933876    2556 manager.go:216] Machine: {Timestamp:2026-07-09 18:53:09.932621005 +0000 UTC m=+0.285056060 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100713 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2335a36f78ff424af596bed2a08f15 SystemUUID:ec2335a3-6f78-ff42-4af5-96bed2a08f15 BootID:cb81a9f7-4654-4fe0-88fc-71d1c1ccc3c3 Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:31:7e:12:41:53 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:31:7e:12:41:53 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:ae:fc:0a:2e:7d Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.934024    2556 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.934138    2556 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.936748    2556 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.936780    2556 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-93.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.937789    2556 topology_manager.go:138] "Creating topology manager with none policy"
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.937802    2556 container_manager_linux.go:309] "Creating device plugin manager"
Jul 09 18:53:10.220316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.938195    2556 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.938441    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.938733    2556 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.939555    2556 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943104    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943229    2556 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943262    2556 kubelet.go:486] "Attempting to sync node with API server"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943280    2556 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943294    2556 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943303    2556 kubelet.go:398] "Adding apiserver pod source"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943313    2556 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943702    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.943718    2556 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.946077    2556 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.947402    2556 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.947429    2556 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948068    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948082    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948088    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948094    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948102    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948108    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948116    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948122    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948128    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948133    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948143    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948148    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948772    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.948779    2556 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.953752    2556 server.go:1268] "Started kubelet"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.954167    2556 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.955999    2556 server.go:310] "Adding debug handlers to kubelet server"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.956812    2556 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.956855    2556 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.958236    2556 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.959993    2556 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.960321    2556 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.961475    2556 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.961502    2556 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.964259    2556 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.964804    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.965699    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.965826    2556 reconstruct.go:97] "Volume reconstruction finished"
Jul 09 18:53:10.221003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.965839    2556 reconciler.go:29] "Reconciler: start to sync state"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.965910    2556 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-93.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.966201    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.967935    2556 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-93.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.968009    2556 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.968073    2556 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.966164    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-93.ec2.internal.18c0b3d75a575638  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-93.ec2.internal,UID:ip-10-0-129-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-93.ec2.internal,},FirstTimestamp:2026-07-09 18:53:09.953730104 +0000 UTC m=+0.306165160,LastTimestamp:2026-07-09 18:53:09.953730104 +0000 UTC m=+0.306165160,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-93.ec2.internal,}"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.969088    2556 factory.go:55] Registering systemd factory
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.969124    2556 factory.go:223] Registration of the systemd container factory successfully
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.970281    2556 factory.go:153] Registering CRI-O factory
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.970296    2556 factory.go:223] Registration of the crio container factory successfully
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.970361    2556 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.970387    2556 factory.go:103] Registering Raw factory
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.970404    2556 manager.go:1196] Started watching for new ooms in manager
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.970830    2556 manager.go:349] Starting recovery of all containers
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.983927    2556 manager.go:354] Recovery completed
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.988145    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.990164    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.990192    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.990203    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.990769    2556 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.990779    2556 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.990796    2556 state_mem.go:36] "Initialized new in-memory state store"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.990810    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.991552    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.991580    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:53:10.225304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.991598    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:09.992372    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-93.ec2.internal.18c0b3d75c838077  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-93.ec2.internal,UID:ip-10-0-129-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-93.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-93.ec2.internal,},FirstTimestamp:2026-07-09 18:53:09.990178935 +0000 UTC m=+0.342613990,LastTimestamp:2026-07-09 18:53:09.990178935 +0000 UTC m=+0.342613990,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-93.ec2.internal,}"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.992776    2556 policy_none.go:49] "None policy: Start"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.992788    2556 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.992801    2556 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:09.994624    2556 policy_none.go:47] "Start"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.002354    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6j8nj"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.004336    2556 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-93.ec2.internal.18c0b3d75c83c7a1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-93.ec2.internal,UID:ip-10-0-129-93.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-93.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-93.ec2.internal,},FirstTimestamp:2026-07-09 18:53:09.990197153 +0000 UTC m=+0.342632212,LastTimestamp:2026-07-09 18:53:09.990197153 +0000 UTC m=+0.342632212,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-93.ec2.internal,}"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.007657    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-6j8nj"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.031132    2556 manager.go:337] "Starting Device Plugin manager"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.031164    2556 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.031174    2556 server.go:87] "Starting device plugin registration server"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.031448    2556 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.031458    2556 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.031593    2556 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.031669    2556 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.031678    2556 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.032171    2556 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.032215    2556 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.076561    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.078030    2556 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.078071    2556 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.078100    2556 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.078138    2556 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.081588    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.132638    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.133924    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:53:10.433487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.133962    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.133977    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.134008    2556 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.145470    2556 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.145497    2556 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-93.ec2.internal\": node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.178490    2556 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal"]
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.178597    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.179747    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.180307    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.180338    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.180348    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.183468    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.183602    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.183638    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.184227    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.184265    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.184279    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.184344    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.184369    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.184380    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.187520    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.187566    2556 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.188368    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientMemory"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.188409    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasNoDiskPressure"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.188424    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeHasSufficientPID"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.241239    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-93.ec2.internal\" not found" node="ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.246445    2556 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-93.ec2.internal\" not found" node="ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.267315    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50ac5346786c8f7d0f6951a8eaeafe6e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-93.ec2.internal\" (UID: \"50ac5346786c8f7d0f6951a8eaeafe6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.435412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.267342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fca2b0fb057877bc56917835668949e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal\" (UID: \"fca2b0fb057877bc56917835668949e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.267362    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fca2b0fb057877bc56917835668949e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal\" (UID: \"fca2b0fb057877bc56917835668949e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.280160    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.368311    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50ac5346786c8f7d0f6951a8eaeafe6e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-93.ec2.internal\" (UID: \"50ac5346786c8f7d0f6951a8eaeafe6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.368345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fca2b0fb057877bc56917835668949e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal\" (UID: \"fca2b0fb057877bc56917835668949e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.368365    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fca2b0fb057877bc56917835668949e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal\" (UID: \"fca2b0fb057877bc56917835668949e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.368418    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/50ac5346786c8f7d0f6951a8eaeafe6e-config\") pod \"kube-apiserver-proxy-ip-10-0-129-93.ec2.internal\" (UID: \"50ac5346786c8f7d0f6951a8eaeafe6e\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.368473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/fca2b0fb057877bc56917835668949e5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal\" (UID: \"fca2b0fb057877bc56917835668949e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.368498    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/fca2b0fb057877bc56917835668949e5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal\" (UID: \"fca2b0fb057877bc56917835668949e5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.436805 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.380280    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.712421 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.481297    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.712421 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.548605    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.712421 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.549768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:10.712421 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.581828    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:10.712421 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.681923    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:11.007991 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.782959    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:11.007991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.853998    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:53:11.007991 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.883782    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:11.007991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.904882    2556 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 09 18:53:11.007991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.905051    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:53:11.007991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:10.905092    2556 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 09 18:53:11.007991 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:10.984835    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:11.016503 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.012480    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-08 18:48:10 +0000 UTC" deadline="2028-01-05 13:29:03.836494081 +0000 UTC"
Jul 09 18:53:11.016503 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.012524    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13074h35m52.823974053s"
Jul 09 18:53:11.217909 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:11.085801    2556 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-93.ec2.internal\" not found"
Jul 09 18:53:11.217909 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.091277    2556 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:53:11.217909 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.103000    2556 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:53:11.217909 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.161996    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:11.217909 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.180441    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:53:11.217909 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.181341    2556 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal"
Jul 09 18:53:11.217909 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.188862    2556 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 09 18:53:11.343467 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:11.234268    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfca2b0fb057877bc56917835668949e5.slice/crio-6474bf5e0f9656d27aa5d16c4d730d0346608cd03ce2e34d225103ad53888e5a WatchSource:0}: Error finding container 6474bf5e0f9656d27aa5d16c4d730d0346608cd03ce2e34d225103ad53888e5a: Status 404 returned error can't find the container with id 6474bf5e0f9656d27aa5d16c4d730d0346608cd03ce2e34d225103ad53888e5a
Jul 09 18:53:11.343467 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:11.234491    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ac5346786c8f7d0f6951a8eaeafe6e.slice/crio-119c4b61b9cc5b4ff6fa96a494bfe8ed9b889655e7c4d6e3fee1ba10d51af6f0 WatchSource:0}: Error finding container 119c4b61b9cc5b4ff6fa96a494bfe8ed9b889655e7c4d6e3fee1ba10d51af6f0: Status 404 returned error can't find the container with id 119c4b61b9cc5b4ff6fa96a494bfe8ed9b889655e7c4d6e3fee1ba10d51af6f0
Jul 09 18:53:11.343467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.238887    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.946742    2556 apiserver.go:52] "Watching apiserver"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.954594    2556 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.955777    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-2qw8v","openshift-network-diagnostics/network-check-target-rs466","openshift-network-operator/iptables-alerter-4xkfn","kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal","openshift-dns/node-resolver-jkf4m","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal","openshift-multus/multus-additional-cni-plugins-v4qmq","openshift-multus/network-metrics-daemon-66mk7","openshift-ovn-kubernetes/ovnkube-node-kbcj5","kube-system/global-pull-secret-syncer-v9667","kube-system/konnectivity-agent-9gvq9","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r","openshift-cluster-node-tuning-operator/tuned-25c92","openshift-image-registry/node-ca-9qhwv"]
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.958433    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:11.958525    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.960763    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.961727    2556 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.963593    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.963918    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-fctnq\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.965061    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.965390    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.966498    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.966931    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-rwldj\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.967442    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.967556    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.967848    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.969657    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.969700    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.970649    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-qmfjd\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.970831    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.970835    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.972158    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:11.972224    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.972235    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.972354    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-wdmng\""
Jul 09 18:53:12.029190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.972459    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.974755    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.975725    2556 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.977205    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.977280    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.977320    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-lmw5k\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.977616    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.977801    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978039    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978091    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-kubernetes\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978129    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-systemd\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978183    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-run\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-sys\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-var-lib-kubelet\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978250    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqflr\" (UniqueName: \"kubernetes.io/projected/994e20d2-bcfe-41a3-8aee-83274fae2962-kube-api-access-jqflr\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978269    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvj7p\" (UniqueName: \"kubernetes.io/projected/b615b4b1-3519-43ed-bdaa-8c465dd97a48-kube-api-access-nvj7p\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978289    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.031088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978313    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysconfig\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978332    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-host\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978351    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b615b4b1-3519-43ed-bdaa-8c465dd97a48-iptables-alerter-script\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978374    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b67611f8-3dcd-4a7f-8948-3ea4b324e439-serviceca\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978422    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hbrwc\" (UniqueName: \"kubernetes.io/projected/b67611f8-3dcd-4a7f-8948-3ea4b324e439-kube-api-access-hbrwc\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978452    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-modprobe-d\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978477    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysctl-d\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978493    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-lib-modules\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/994e20d2-bcfe-41a3-8aee-83274fae2962-tmp\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978534    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f721565e-88df-4cad-b2e4-49193dcfbddd-kubelet-config\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978572    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysctl-conf\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978587    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-tuned\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978644    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b615b4b1-3519-43ed-bdaa-8c465dd97a48-host-slash\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b67611f8-3dcd-4a7f-8948-3ea4b324e439-host\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.034984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.978685    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f721565e-88df-4cad-b2e4-49193dcfbddd-dbus\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.980019    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:11.980074    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.980493    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.980721    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.981212    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-t6q6f\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.981238    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.981260    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.981312    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.982573    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.985082    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.985177    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.985187    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-kcvxf\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.987508    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.988075    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.988120    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.988150    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-zwhcx\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.990522    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.990524    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-jt29m\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.995437    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:11.995450    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.029272    2556 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz276"
Jul 09 18:53:12.035764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.032206    2556 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-zz276"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.065572    2556 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-kubernetes\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079291    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvj7p\" (UniqueName: \"kubernetes.io/projected/b615b4b1-3519-43ed-bdaa-8c465dd97a48-kube-api-access-nvj7p\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-var-lib-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079377    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-kubernetes\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079386    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-k8s-cni-cncf-io\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-netns\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079520    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-cni-bin\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079566    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-device-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-sys-fs\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079656    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079692    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-os-release\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.142860 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079724    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-multus-certs\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079751    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vx44\" (UniqueName: \"kubernetes.io/projected/58622854-61de-4acb-a671-1cd2aa16fc97-kube-api-access-6vx44\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079782    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b67611f8-3dcd-4a7f-8948-3ea4b324e439-host\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079809    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079833    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-cni-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079857    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b67611f8-3dcd-4a7f-8948-3ea4b324e439-host\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079915    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079934    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovnkube-config\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079957    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-systemd\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.079991    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-sys\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080007    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqflr\" (UniqueName: \"kubernetes.io/projected/994e20d2-bcfe-41a3-8aee-83274fae2962-kube-api-access-jqflr\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080384    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b67611f8-3dcd-4a7f-8948-3ea4b324e439-serviceca\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080435    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-cni-netd\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.143649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080450    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-sys\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080491    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-registration-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080387    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-systemd\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080551    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080629    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysconfig\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080682    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysconfig\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080721    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-slash\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080764    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-ovn\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080801    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080858    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zg7gw\" (UniqueName: \"kubernetes.io/projected/a8a09070-edcc-43d2-8b08-a3fd1950f284-kube-api-access-zg7gw\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080894    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-socket-dir-parent\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.080944    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-modprobe-d\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081066    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-modprobe-d\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081127    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysctl-d\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081185    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/994e20d2-bcfe-41a3-8aee-83274fae2962-tmp\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-kubelet\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081310    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-systemd\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081343    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58622854-61de-4acb-a671-1cd2aa16fc97-hosts-file\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081377    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/b67611f8-3dcd-4a7f-8948-3ea4b324e439-serviceca\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081388    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f721565e-88df-4cad-b2e4-49193dcfbddd-kubelet-config\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081425    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b615b4b1-3519-43ed-bdaa-8c465dd97a48-host-slash\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081490    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-system-cni-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081515    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-etc-kubernetes\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081569    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-os-release\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081620    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f721565e-88df-4cad-b2e4-49193dcfbddd-dbus\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081654    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-run\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/f721565e-88df-4cad-b2e4-49193dcfbddd-kubelet-config\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081787    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/b615b4b1-3519-43ed-bdaa-8c465dd97a48-host-slash\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.145989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081992    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/f721565e-88df-4cad-b2e4-49193dcfbddd-dbus\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082048    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-node-log\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082091    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-hostroot\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082117    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-run\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082119    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q888s\" (UniqueName: \"kubernetes.io/projected/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-kube-api-access-q888s\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082015    2556 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082173    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58622854-61de-4acb-a671-1cd2aa16fc97-tmp-dir\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082221    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e-konnectivity-ca\") pod \"konnectivity-agent-9gvq9\" (UID: \"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e\") " pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082249    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082278    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-host\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.082384    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.082494    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret podName:f721565e-88df-4cad-b2e4-49193dcfbddd nodeName:}" failed. No retries permitted until 2026-07-09 18:53:12.582442349 +0000 UTC m=+2.934877415 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret") pod "global-pull-secret-syncer-v9667" (UID: "f721565e-88df-4cad-b2e4-49193dcfbddd") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082731    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-cni-binary-copy\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082766    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-socket-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.147147 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082795    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-cnibin\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082824    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082858    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-tuned\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082876    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-host\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovn-node-metrics-cert\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.082973    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-cni-multus\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnjkq\" (UniqueName: \"kubernetes.io/projected/5f6e8995-b752-4679-887f-4cd3a9826b7c-kube-api-access-nnjkq\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083035    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-cnibin\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083116    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-var-lib-kubelet\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hbrwc\" (UniqueName: \"kubernetes.io/projected/b67611f8-3dcd-4a7f-8948-3ea4b324e439-kube-api-access-hbrwc\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083181    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-etc-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083207    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-log-socket\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083238    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-daemon-config\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.147920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b615b4b1-3519-43ed-bdaa-8c465dd97a48-iptables-alerter-script\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083304    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-run-netns\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083416    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-cni-bin\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083452    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083481    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-system-cni-dir\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083537    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083571    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwt7x\" (UniqueName: \"kubernetes.io/projected/1ba3f622-358b-4925-a89d-3ac1d413b48e-kube-api-access-wwt7x\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.083956    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-var-lib-kubelet\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.081649    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysctl-d\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.084795    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-lib-modules\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.084863    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-conf-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.084906    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-lib-modules\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.084925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e-agent-certs\") pod \"konnectivity-agent-9gvq9\" (UID: \"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e\") " pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.085016    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysctl-conf\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.149153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.085061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-systemd-units\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.085098    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-env-overrides\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.085130    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovnkube-script-lib\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.085187    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-sysctl-conf\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.085588    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-kubelet\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.085635    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srsr\" (UniqueName: \"kubernetes.io/projected/d8946aaf-6587-4254-8c07-bb2b12cf87d0-kube-api-access-2srsr\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.086336    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/b615b4b1-3519-43ed-bdaa-8c465dd97a48-iptables-alerter-script\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.087121    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/994e20d2-bcfe-41a3-8aee-83274fae2962-tmp\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.087367    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/994e20d2-bcfe-41a3-8aee-83274fae2962-etc-tuned\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.090749    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvj7p\" (UniqueName: \"kubernetes.io/projected/b615b4b1-3519-43ed-bdaa-8c465dd97a48-kube-api-access-nvj7p\") pod \"iptables-alerter-4xkfn\" (UID: \"b615b4b1-3519-43ed-bdaa-8c465dd97a48\") " pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.096851    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqflr\" (UniqueName: \"kubernetes.io/projected/994e20d2-bcfe-41a3-8aee-83274fae2962-kube-api-access-jqflr\") pod \"tuned-25c92\" (UID: \"994e20d2-bcfe-41a3-8aee-83274fae2962\") " pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.097279    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hbrwc\" (UniqueName: \"kubernetes.io/projected/b67611f8-3dcd-4a7f-8948-3ea4b324e439-kube-api-access-hbrwc\") pod \"node-ca-9qhwv\" (UID: \"b67611f8-3dcd-4a7f-8948-3ea4b324e439\") " pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.151120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.148086    2556 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:53:12.186395 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186360    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-cni-binary-copy\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.186522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-socket-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.186522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186431    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-cnibin\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.186522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186459    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.186522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186487    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovn-node-metrics-cert\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.186522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186512    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-cni-multus\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186537    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnjkq\" (UniqueName: \"kubernetes.io/projected/5f6e8995-b752-4679-887f-4cd3a9826b7c-kube-api-access-nnjkq\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186578    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186606    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-cnibin\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186630    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-cnibin\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-etc-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186683    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-etc-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186694    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-log-socket\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-daemon-config\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.186768 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186751    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-run-netns\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186774    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-cni-bin\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186799    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186826    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-system-cni-dir\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186852    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186878    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwt7x\" (UniqueName: \"kubernetes.io/projected/1ba3f622-358b-4925-a89d-3ac1d413b48e-kube-api-access-wwt7x\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186908    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-conf-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186933    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e-agent-certs\") pod \"konnectivity-agent-9gvq9\" (UID: \"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e\") " pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.186985    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-systemd-units\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187012    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-env-overrides\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187044    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovnkube-script-lib\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187067    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-kubelet\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187086    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-cni-binary-copy\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187107    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srsr\" (UniqueName: \"kubernetes.io/projected/d8946aaf-6587-4254-8c07-bb2b12cf87d0-kube-api-access-2srsr\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.187135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-var-lib-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187150    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-cnibin\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187163    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187179    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-run-netns\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187190    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-k8s-cni-cncf-io\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187214    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-kubelet-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187223    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-tuning-conf-dir\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187231    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-log-socket\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187230    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-netns\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187259    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-cni-bin\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187269    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-cni-bin\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-device-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187303    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-cni-multus\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187317    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-sys-fs\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187306    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-system-cni-dir\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.187735 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187356    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-cni-bin\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187353    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-var-lib-kubelet\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187273    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-netns\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187401    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-device-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187446    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-sys-fs\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187510    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-var-lib-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187557    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-openvswitch\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187624    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-k8s-cni-cncf-io\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187671    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-systemd-units\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187771    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-daemon-config\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187839    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-conf-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187878    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187892    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-socket-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-os-release\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-multus-certs\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187959    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-os-release\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.187971    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vx44\" (UniqueName: \"kubernetes.io/projected/58622854-61de-4acb-a671-1cd2aa16fc97-kube-api-access-6vx44\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188027    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-cni-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188057    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.188031    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.188184    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs podName:a8a09070-edcc-43d2-8b08-a3fd1950f284 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:12.688160783 +0000 UTC m=+3.040595839 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs") pod "network-metrics-daemon-66mk7" (UID: "a8a09070-edcc-43d2-8b08-a3fd1950f284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188241    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovnkube-script-lib\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188285    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188313    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovnkube-config\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188347    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-cni-netd\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188375    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-registration-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188403    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188430    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-slash\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.188940 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188454    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-ovn\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188482    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188508    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zg7gw\" (UniqueName: \"kubernetes.io/projected/a8a09070-edcc-43d2-8b08-a3fd1950f284-kube-api-access-zg7gw\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188535    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-socket-dir-parent\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188568    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-kubelet\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188593    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-systemd\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188640    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58622854-61de-4acb-a671-1cd2aa16fc97-hosts-file\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188667    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-system-cni-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188695    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-etc-kubernetes\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188721    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-os-release\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188752    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-node-log\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188777    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-hostroot\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188783    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-env-overrides\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188804    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q888s\" (UniqueName: \"kubernetes.io/projected/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-kube-api-access-q888s\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188858    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-socket-dir-parent\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.189560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188889    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58622854-61de-4acb-a671-1cd2aa16fc97-tmp-dir\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188917    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e-konnectivity-ca\") pod \"konnectivity-agent-9gvq9\" (UID: \"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e\") " pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.188943    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-cni-binary-copy\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189048    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-kubelet\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189074    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-etc-selinux\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189096    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-systemd\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189144    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-multus-cni-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189176    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/58622854-61de-4acb-a671-1cd2aa16fc97-hosts-file\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189189    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-host-run-multus-certs\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189239    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-system-cni-dir\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189252    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/1ba3f622-358b-4925-a89d-3ac1d413b48e-os-release\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189290    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-etc-kubernetes\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189287    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-node-log\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189332    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-hostroot\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189345    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-slash\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190173 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189568    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-run-ovn\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189630    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-run-ovn-kubernetes\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189678    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189567    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e-konnectivity-ca\") pod \"konnectivity-agent-9gvq9\" (UID: \"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e\") " pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/d8946aaf-6587-4254-8c07-bb2b12cf87d0-registration-dir\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189804    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/5f6e8995-b752-4679-887f-4cd3a9826b7c-host-cni-netd\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovnkube-config\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.189986    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/58622854-61de-4acb-a671-1cd2aa16fc97-tmp-dir\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.190212    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e-agent-certs\") pod \"konnectivity-agent-9gvq9\" (UID: \"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e\") " pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.190292    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.190781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.190472    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1ba3f622-358b-4925-a89d-3ac1d413b48e-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.192207 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.192182    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/5f6e8995-b752-4679-887f-4cd3a9826b7c-ovn-node-metrics-cert\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.198207 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.198184    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:53:12.198207 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.198208    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:53:12.198327 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.198222    2556 projected.go:196] Error preparing data for projected volume kube-api-access-hzxl9 for pod openshift-network-diagnostics/network-check-target-rs466: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:12.198327 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.198287    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9 podName:e99ad357-fdc9-4571-9528-e5743907647a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:12.698268687 +0000 UTC m=+3.050703733 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-hzxl9" (UniqueName: "kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9") pod "network-check-target-rs466" (UID: "e99ad357-fdc9-4571-9528-e5743907647a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:12.199718 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.199674    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnjkq\" (UniqueName: \"kubernetes.io/projected/5f6e8995-b752-4679-887f-4cd3a9826b7c-kube-api-access-nnjkq\") pod \"ovnkube-node-kbcj5\" (UID: \"5f6e8995-b752-4679-887f-4cd3a9826b7c\") " pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.201557 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.201511    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vx44\" (UniqueName: \"kubernetes.io/projected/58622854-61de-4acb-a671-1cd2aa16fc97-kube-api-access-6vx44\") pod \"node-resolver-jkf4m\" (UID: \"58622854-61de-4acb-a671-1cd2aa16fc97\") " pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.202919 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.202895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwt7x\" (UniqueName: \"kubernetes.io/projected/1ba3f622-358b-4925-a89d-3ac1d413b48e-kube-api-access-wwt7x\") pod \"multus-additional-cni-plugins-v4qmq\" (UID: \"1ba3f622-358b-4925-a89d-3ac1d413b48e\") " pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.202997 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.202947    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srsr\" (UniqueName: \"kubernetes.io/projected/d8946aaf-6587-4254-8c07-bb2b12cf87d0-kube-api-access-2srsr\") pod \"aws-ebs-csi-driver-node-twb8r\" (UID: \"d8946aaf-6587-4254-8c07-bb2b12cf87d0\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.203236 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.203213    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q888s\" (UniqueName: \"kubernetes.io/projected/9cf25dc3-b45e-4228-b431-74a01ff7ac2f-kube-api-access-q888s\") pod \"multus-2qw8v\" (UID: \"9cf25dc3-b45e-4228-b431-74a01ff7ac2f\") " pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.204027 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.204006    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zg7gw\" (UniqueName: \"kubernetes.io/projected/a8a09070-edcc-43d2-8b08-a3fd1950f284-kube-api-access-zg7gw\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:12.214411 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.214394    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-4xkfn"
Jul 09 18:53:12.229040 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.228979    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal" event={"ID":"50ac5346786c8f7d0f6951a8eaeafe6e","Type":"ContainerStarted","Data":"119c4b61b9cc5b4ff6fa96a494bfe8ed9b889655e7c4d6e3fee1ba10d51af6f0"}
Jul 09 18:53:12.230479 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.230450    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal" event={"ID":"fca2b0fb057877bc56917835668949e5","Type":"ContainerStarted","Data":"6474bf5e0f9656d27aa5d16c4d730d0346608cd03ce2e34d225103ad53888e5a"}
Jul 09 18:53:12.235161 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.235131    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb615b4b1_3519_43ed_bdaa_8c465dd97a48.slice/crio-3a80ba56a40ab9898d96943083f1108f4296d0f0f5457017500df3017003a9ab WatchSource:0}: Error finding container 3a80ba56a40ab9898d96943083f1108f4296d0f0f5457017500df3017003a9ab: Status 404 returned error can't find the container with id 3a80ba56a40ab9898d96943083f1108f4296d0f0f5457017500df3017003a9ab
Jul 09 18:53:12.249278 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.249255    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9qhwv"
Jul 09 18:53:12.261032 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.260943    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-jkf4m"
Jul 09 18:53:12.262569 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.262538    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb67611f8_3dcd_4a7f_8948_3ea4b324e439.slice/crio-c3b028715db764a666c6f88570c919b1907224cb279f478dbeb17656a6210eb4 WatchSource:0}: Error finding container c3b028715db764a666c6f88570c919b1907224cb279f478dbeb17656a6210eb4: Status 404 returned error can't find the container with id c3b028715db764a666c6f88570c919b1907224cb279f478dbeb17656a6210eb4
Jul 09 18:53:12.272079 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.272048    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58622854_61de_4acb_a671_1cd2aa16fc97.slice/crio-8b877a896bdd37cea27ec66e86a1fafdf550daa5e82d40ef3f18ad022f2c3cda WatchSource:0}: Error finding container 8b877a896bdd37cea27ec66e86a1fafdf550daa5e82d40ef3f18ad022f2c3cda: Status 404 returned error can't find the container with id 8b877a896bdd37cea27ec66e86a1fafdf550daa5e82d40ef3f18ad022f2c3cda
Jul 09 18:53:12.273197 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.273176    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-25c92"
Jul 09 18:53:12.280508 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.280451    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:12.290573 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.288757    2556 log.go:32] "PullImage from image service failed" err="rpc error: code = Unknown desc = unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6"
Jul 09 18:53:12.290573 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.288938    2556 kuberuntime_manager.go:1459] "Unhandled Error" err="init container setup start failed in pod kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal_openshift-machine-config-operator(fca2b0fb057877bc56917835668949e5): ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image" logger="UnhandledError"
Jul 09 18:53:12.290573 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.288983    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod994e20d2_bcfe_41a3_8aee_83274fae2962.slice/crio-199e61293da9573204b861ef5d67fe9a3e87c33bd0c551d7780d04a8490bc787 WatchSource:0}: Error finding container 199e61293da9573204b861ef5d67fe9a3e87c33bd0c551d7780d04a8490bc787: Status 404 returned error can't find the container with id 199e61293da9573204b861ef5d67fe9a3e87c33bd0c551d7780d04a8490bc787
Jul 09 18:53:12.290573 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.288984    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with ErrImagePull: \"unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal" podUID="fca2b0fb057877bc56917835668949e5"
Jul 09 18:53:12.295960 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.295927    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-v4qmq"
Jul 09 18:53:12.296220 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.296197    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f6e8995_b752_4679_887f_4cd3a9826b7c.slice/crio-738071eb08f480b1868606ed2d8b9f3f21d3f6c409aced6a61f9f61b042c7953 WatchSource:0}: Error finding container 738071eb08f480b1868606ed2d8b9f3f21d3f6c409aced6a61f9f61b042c7953: Status 404 returned error can't find the container with id 738071eb08f480b1868606ed2d8b9f3f21d3f6c409aced6a61f9f61b042c7953
Jul 09 18:53:12.307302 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.307281    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-2qw8v"
Jul 09 18:53:12.309209 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.309178    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ba3f622_358b_4925_a89d_3ac1d413b48e.slice/crio-9090f9988306cf20b7af01c73ce8639ff1724b6d8fe6c42d902c421a0e825899 WatchSource:0}: Error finding container 9090f9988306cf20b7af01c73ce8639ff1724b6d8fe6c42d902c421a0e825899: Status 404 returned error can't find the container with id 9090f9988306cf20b7af01c73ce8639ff1724b6d8fe6c42d902c421a0e825899
Jul 09 18:53:12.316785 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.316548    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:12.322287 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.321699    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cf25dc3_b45e_4228_b431_74a01ff7ac2f.slice/crio-65f45e8185239df50a31c2af3d4291c497d3d213fe1550070f0c519cf22c4cf7 WatchSource:0}: Error finding container 65f45e8185239df50a31c2af3d4291c497d3d213fe1550070f0c519cf22c4cf7: Status 404 returned error can't find the container with id 65f45e8185239df50a31c2af3d4291c497d3d213fe1550070f0c519cf22c4cf7
Jul 09 18:53:12.326202 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.325879    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r"
Jul 09 18:53:12.339338 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:12.339304    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8946aaf_6587_4254_8c07_bb2b12cf87d0.slice/crio-a3f3ae5b5b3e648cd9a6b81a075f9f66c54b8eccb7c7db9401b1976b11bdb703 WatchSource:0}: Error finding container a3f3ae5b5b3e648cd9a6b81a075f9f66c54b8eccb7c7db9401b1976b11bdb703: Status 404 returned error can't find the container with id a3f3ae5b5b3e648cd9a6b81a075f9f66c54b8eccb7c7db9401b1976b11bdb703
Jul 09 18:53:12.577720 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.577615    2556 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 09 18:53:12.593461 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.593417    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:12.593669 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.593605    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:12.593743 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.593687    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret podName:f721565e-88df-4cad-b2e4-49193dcfbddd nodeName:}" failed. No retries permitted until 2026-07-09 18:53:13.593661831 +0000 UTC m=+3.946096891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret") pod "global-pull-secret-syncer-v9667" (UID: "f721565e-88df-4cad-b2e4-49193dcfbddd") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:12.693990 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.693795    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:12.693990 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.693976    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:12.694201 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.694080    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs podName:a8a09070-edcc-43d2-8b08-a3fd1950f284 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:13.694042228 +0000 UTC m=+4.046477287 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs") pod "network-metrics-daemon-66mk7" (UID: "a8a09070-edcc-43d2-8b08-a3fd1950f284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:12.795306 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:12.795272    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:12.795492 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.795469    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:53:12.795532 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.795499    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:53:12.795532 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.795510    2556 projected.go:196] Error preparing data for projected volume kube-api-access-hzxl9 for pod openshift-network-diagnostics/network-check-target-rs466: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:12.795650 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:12.795580    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9 podName:e99ad357-fdc9-4571-9528-e5743907647a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:13.795561231 +0000 UTC m=+4.147996275 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzxl9" (UniqueName: "kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9") pod "network-check-target-rs466" (UID: "e99ad357-fdc9-4571-9528-e5743907647a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:13.035276 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.035168    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:48:12 +0000 UTC" deadline="2028-02-02 23:34:37.541621875 +0000 UTC"
Jul 09 18:53:13.035276 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.035222    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13756h41m24.50640217s"
Jul 09 18:53:13.235944 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.234454    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2qw8v" event={"ID":"9cf25dc3-b45e-4228-b431-74a01ff7ac2f","Type":"ContainerStarted","Data":"65f45e8185239df50a31c2af3d4291c497d3d213fe1550070f0c519cf22c4cf7"}
Jul 09 18:53:13.238328 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.238291    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-25c92" event={"ID":"994e20d2-bcfe-41a3-8aee-83274fae2962","Type":"ContainerStarted","Data":"199e61293da9573204b861ef5d67fe9a3e87c33bd0c551d7780d04a8490bc787"}
Jul 09 18:53:13.241559 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.240951    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9qhwv" event={"ID":"b67611f8-3dcd-4a7f-8948-3ea4b324e439","Type":"ContainerStarted","Data":"c3b028715db764a666c6f88570c919b1907224cb279f478dbeb17656a6210eb4"}
Jul 09 18:53:13.246882 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.246847    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xkfn" event={"ID":"b615b4b1-3519-43ed-bdaa-8c465dd97a48","Type":"ContainerStarted","Data":"3a80ba56a40ab9898d96943083f1108f4296d0f0f5457017500df3017003a9ab"}
Jul 09 18:53:13.250921 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.250871    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9gvq9" event={"ID":"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e","Type":"ContainerStarted","Data":"6cb082f4450857b09a93096e8f5dbb59089ceaa425c3e047b082fc5010277635"}
Jul 09 18:53:13.254951 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.254920    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerStarted","Data":"9090f9988306cf20b7af01c73ce8639ff1724b6d8fe6c42d902c421a0e825899"}
Jul 09 18:53:13.258324 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.258238    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"738071eb08f480b1868606ed2d8b9f3f21d3f6c409aced6a61f9f61b042c7953"}
Jul 09 18:53:13.259574 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.259530    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jkf4m" event={"ID":"58622854-61de-4acb-a671-1cd2aa16fc97","Type":"ContainerStarted","Data":"8b877a896bdd37cea27ec66e86a1fafdf550daa5e82d40ef3f18ad022f2c3cda"}
Jul 09 18:53:13.262226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.261678    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r" event={"ID":"d8946aaf-6587-4254-8c07-bb2b12cf87d0","Type":"ContainerStarted","Data":"a3f3ae5b5b3e648cd9a6b81a075f9f66c54b8eccb7c7db9401b1976b11bdb703"}
Jul 09 18:53:13.263204 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.262875    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"setup\" with ImagePullBackOff: \"Back-off pulling image \\\"quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:9d1573db16143027c8ffaf46a01ce12f2bcd89c15410c42ca54be4b3b1ddfdd6\\\": ErrImagePull: unable to pull image or OCI artifact: pull image err: copying system image from manifest list: parsing image configuration: fetching blob: received unexpected HTTP status: 503 Service Unavailable; artifact err: provided artifact is a container image\"" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal" podUID="fca2b0fb057877bc56917835668949e5"
Jul 09 18:53:13.602914 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.602221    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:13.602914 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.602441    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:13.602914 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.602509    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret podName:f721565e-88df-4cad-b2e4-49193dcfbddd nodeName:}" failed. No retries permitted until 2026-07-09 18:53:15.602490172 +0000 UTC m=+5.954925233 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret") pod "global-pull-secret-syncer-v9667" (UID: "f721565e-88df-4cad-b2e4-49193dcfbddd") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:13.703417 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.703373    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:13.704132 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.703624    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:13.704132 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.703709    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs podName:a8a09070-edcc-43d2-8b08-a3fd1950f284 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:15.70368747 +0000 UTC m=+6.056122532 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs") pod "network-metrics-daemon-66mk7" (UID: "a8a09070-edcc-43d2-8b08-a3fd1950f284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:13.805333 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:13.804582    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:13.805333 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.804770    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:53:13.805333 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.804795    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:53:13.805333 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.804809    2556 projected.go:196] Error preparing data for projected volume kube-api-access-hzxl9 for pod openshift-network-diagnostics/network-check-target-rs466: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:13.805333 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:13.804872    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9 podName:e99ad357-fdc9-4571-9528-e5743907647a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:15.804853121 +0000 UTC m=+6.157288171 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzxl9" (UniqueName: "kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9") pod "network-check-target-rs466" (UID: "e99ad357-fdc9-4571-9528-e5743907647a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:14.037495 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:14.037086    2556 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-08 18:48:12 +0000 UTC" deadline="2028-02-27 19:07:22.089797021 +0000 UTC"
Jul 09 18:53:14.037495 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:14.037244    2556 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14352h14m8.052589771s"
Jul 09 18:53:14.083640 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:14.080450    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:14.083640 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:14.080656    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:14.083640 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:14.081175    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:14.083640 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:14.081299    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:14.083640 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:14.081517    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:14.083640 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:14.081645    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:14.277935 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:14.277886    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal" event={"ID":"50ac5346786c8f7d0f6951a8eaeafe6e","Type":"ContainerStarted","Data":"62f1bb95345a1852c491dfd66e7152da8fcd5e6a1f002b6af8e3bae1fe0fa8cd"}
Jul 09 18:53:15.619971 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:15.619932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:15.620442 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.620112    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:15.620442 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.620176    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret podName:f721565e-88df-4cad-b2e4-49193dcfbddd nodeName:}" failed. No retries permitted until 2026-07-09 18:53:19.620157291 +0000 UTC m=+9.972592351 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret") pod "global-pull-secret-syncer-v9667" (UID: "f721565e-88df-4cad-b2e4-49193dcfbddd") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:15.720698 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:15.720631    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:15.721559 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.720931    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:15.721559 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.721008    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs podName:a8a09070-edcc-43d2-8b08-a3fd1950f284 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:19.720989087 +0000 UTC m=+10.073424133 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs") pod "network-metrics-daemon-66mk7" (UID: "a8a09070-edcc-43d2-8b08-a3fd1950f284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:15.821555 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:15.821502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:15.821771 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.821753    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:53:15.821848 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.821780    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:53:15.821848 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.821793    2556 projected.go:196] Error preparing data for projected volume kube-api-access-hzxl9 for pod openshift-network-diagnostics/network-check-target-rs466: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:15.821945 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:15.821852    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9 podName:e99ad357-fdc9-4571-9528-e5743907647a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:19.821834069 +0000 UTC m=+10.174269117 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzxl9" (UniqueName: "kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9") pod "network-check-target-rs466" (UID: "e99ad357-fdc9-4571-9528-e5743907647a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:16.079688 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:16.079237    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:16.079688 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:16.079259    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:16.079688 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:16.079408    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:16.079688 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:16.079454    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:16.079688 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:16.079505    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:16.079688 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:16.079593    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:18.078825 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:18.078789    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:18.079243 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:18.078912    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:18.079243 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:18.079164    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:18.079243 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:18.079201    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:18.079417 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:18.079282    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:18.079417 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:18.079350    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:19.653937 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:19.653890    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:19.654388 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.654069    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:19.654388 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.654151    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret podName:f721565e-88df-4cad-b2e4-49193dcfbddd nodeName:}" failed. No retries permitted until 2026-07-09 18:53:27.654126925 +0000 UTC m=+18.006561973 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret") pod "global-pull-secret-syncer-v9667" (UID: "f721565e-88df-4cad-b2e4-49193dcfbddd") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:19.755278 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:19.755244    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:19.755445 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.755397    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:19.755499 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.755456    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs podName:a8a09070-edcc-43d2-8b08-a3fd1950f284 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:27.755441621 +0000 UTC m=+18.107876665 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs") pod "network-metrics-daemon-66mk7" (UID: "a8a09070-edcc-43d2-8b08-a3fd1950f284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:19.856772 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:19.856677    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:19.856950 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.856873    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:53:19.856950 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.856900    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:53:19.856950 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.856914    2556 projected.go:196] Error preparing data for projected volume kube-api-access-hzxl9 for pod openshift-network-diagnostics/network-check-target-rs466: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:19.857100 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:19.856982    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9 podName:e99ad357-fdc9-4571-9528-e5743907647a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:27.856961387 +0000 UTC m=+18.209396433 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzxl9" (UniqueName: "kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9") pod "network-check-target-rs466" (UID: "e99ad357-fdc9-4571-9528-e5743907647a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:20.079803 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:20.079723    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:20.080068 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:20.079795    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:20.080208 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:20.079828    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:20.080208 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:20.080137    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:20.080341 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:20.080222    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:20.080341 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:20.080311    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:22.079006 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:22.078977    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:22.079425 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:22.078981    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:22.079425 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:22.079099    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:22.079425 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:22.079126    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:22.079425 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:22.079223    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:22.079425 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:22.079298    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:24.078741 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:24.078704    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:24.079157 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:24.078704    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:24.079157 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:24.078841    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:24.079157 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:24.078934    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:24.079157 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:24.078704    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:24.079157 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:24.079046    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:25.096080 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:25.096020    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-93.ec2.internal" podStartSLOduration=14.095999847 podStartE2EDuration="14.095999847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:14.302594255 +0000 UTC m=+4.655029317" watchObservedRunningTime="2026-07-09 18:53:25.095999847 +0000 UTC m=+15.448434916"
Jul 09 18:53:26.078827 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:26.078793    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:26.078827 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:26.078825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:26.079070 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:26.078793    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:26.079070 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:26.078925    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:26.079159 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:26.079077    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:26.079159 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:26.079141    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:27.717199 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:27.717157    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:27.717737 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.717320    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:27.717737 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.717384    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret podName:f721565e-88df-4cad-b2e4-49193dcfbddd nodeName:}" failed. No retries permitted until 2026-07-09 18:53:43.717369059 +0000 UTC m=+34.069804102 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret") pod "global-pull-secret-syncer-v9667" (UID: "f721565e-88df-4cad-b2e4-49193dcfbddd") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:27.818327 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:27.818289    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:27.818510 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.818443    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:27.818592 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.818516    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs podName:a8a09070-edcc-43d2-8b08-a3fd1950f284 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:43.818493719 +0000 UTC m=+34.170928765 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs") pod "network-metrics-daemon-66mk7" (UID: "a8a09070-edcc-43d2-8b08-a3fd1950f284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:27.919142 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:27.919097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:27.919300 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.919282    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:53:27.919338 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.919304    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:53:27.919338 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.919315    2556 projected.go:196] Error preparing data for projected volume kube-api-access-hzxl9 for pod openshift-network-diagnostics/network-check-target-rs466: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:27.919426 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:27.919363    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9 podName:e99ad357-fdc9-4571-9528-e5743907647a nodeName:}" failed. No retries permitted until 2026-07-09 18:53:43.919350503 +0000 UTC m=+34.271785546 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzxl9" (UniqueName: "kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9") pod "network-check-target-rs466" (UID: "e99ad357-fdc9-4571-9528-e5743907647a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:28.078374 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:28.078289    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:28.078567 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:28.078289    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:28.078567 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:28.078421    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:28.078567 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:28.078289    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:28.078567 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:28.078499    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:28.078798 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:28.078605    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:30.081215 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:30.081181    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:30.081721 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:30.081289    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:30.081721 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:30.081385    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:30.081721 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:30.081497    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:30.081721 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:30.081548    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:30.081721 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:30.081603    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:32.079259 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.078501    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:32.079259 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:32.078906    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:32.079259 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.078996    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:32.079259 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:32.079068    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:32.079259 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.079154    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:32.079259 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:32.079230    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:32.319781 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.319750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-9gvq9" event={"ID":"48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e","Type":"ContainerStarted","Data":"3fd097f0ccee3a6a541d0734e10f3ce6816b14071729e356ad2f8c81a2fc7a25"}
Jul 09 18:53:32.321290 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.321269    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerStarted","Data":"fd048617350db57702c499399056ff76374d74dcb97da33d2c378621a623f34b"}
Jul 09 18:53:32.323669 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.323645    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"2f47feba310b021b52c4b554bae20902bb09d315edb5fcbe6b3d23a5f9b950d5"}
Jul 09 18:53:32.323755 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.323718    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"238213899ca593f65ef3e1d659dd0c3443dea5b1ce4094493f7db734f01fa43d"}
Jul 09 18:53:32.325210 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.325160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-jkf4m" event={"ID":"58622854-61de-4acb-a671-1cd2aa16fc97","Type":"ContainerStarted","Data":"279b78bfae834f5a58ede15300f6618b55867977075aad1a7e313563553b9b4c"}
Jul 09 18:53:32.327027 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.326980    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r" event={"ID":"d8946aaf-6587-4254-8c07-bb2b12cf87d0","Type":"ContainerStarted","Data":"1a5b6166876009ecb798706d580c590c7d848381ad858fc10d028546522c2cd2"}
Jul 09 18:53:32.331984 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.331953    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-2qw8v" event={"ID":"9cf25dc3-b45e-4228-b431-74a01ff7ac2f","Type":"ContainerStarted","Data":"e6aaca27be856c1489741c1caa479e65f912a8c3b0d428ea500bf868c402fff2"}
Jul 09 18:53:32.333672 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.333646    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-25c92" event={"ID":"994e20d2-bcfe-41a3-8aee-83274fae2962","Type":"ContainerStarted","Data":"09f9a13c45c304b87912397821f7071c69fb9be80730b936d210cdf1284bc1ff"}
Jul 09 18:53:32.335054 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.335028    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9qhwv" event={"ID":"b67611f8-3dcd-4a7f-8948-3ea4b324e439","Type":"ContainerStarted","Data":"a3e088a7cf9afd198ddee23384c2b257a418d707cdfe25056a5c7218d0dc9276"}
Jul 09 18:53:32.336446 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.336413    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-9gvq9" podStartSLOduration=7.737044411 podStartE2EDuration="22.33640278s" totalImagesPullingTime="14.599358369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:32.336369292 +0000 UTC m=+22.688804369" watchObservedRunningTime="2026-07-09 18:53:32.33640278 +0000 UTC m=+22.688837845"
Jul 09 18:53:32.359303 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.359254    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-jkf4m" podStartSLOduration=2.992855572 podStartE2EDuration="22.359236323s" totalImagesPullingTime="19.366380751s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:32.358723645 +0000 UTC m=+22.711158714" watchObservedRunningTime="2026-07-09 18:53:32.359236323 +0000 UTC m=+22.711671389"
Jul 09 18:53:32.376900 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.376856    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-25c92" podStartSLOduration=3.026590103 podStartE2EDuration="22.37684121s" totalImagesPullingTime="19.350251107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:32.376463293 +0000 UTC m=+22.728898357" watchObservedRunningTime="2026-07-09 18:53:32.37684121 +0000 UTC m=+22.729276275"
Jul 09 18:53:32.418679 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:32.418637    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-2qw8v" podStartSLOduration=3.099090249 podStartE2EDuration="22.418620364s" totalImagesPullingTime="19.319530115s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:32.418423288 +0000 UTC m=+22.770858353" watchObservedRunningTime="2026-07-09 18:53:32.418620364 +0000 UTC m=+22.771055431"
Jul 09 18:53:33.338911 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.338680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-4xkfn" event={"ID":"b615b4b1-3519-43ed-bdaa-8c465dd97a48","Type":"ContainerStarted","Data":"cb4c7481541ac32d5c34f98374af6833aeae9a3a0172bca80dc78ecf6df926dc"}
Jul 09 18:53:33.339996 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.339975    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ba3f622-358b-4925-a89d-3ac1d413b48e" containerID="fd048617350db57702c499399056ff76374d74dcb97da33d2c378621a623f34b" exitCode=0
Jul 09 18:53:33.340079 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.340038    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerDied","Data":"fd048617350db57702c499399056ff76374d74dcb97da33d2c378621a623f34b"}
Jul 09 18:53:33.341481 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.341466    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 18:53:33.341846 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.341827    2556 generic.go:356] "Generic (PLEG): container finished" podID="5f6e8995-b752-4679-887f-4cd3a9826b7c" containerID="2f47feba310b021b52c4b554bae20902bb09d315edb5fcbe6b3d23a5f9b950d5" exitCode=1
Jul 09 18:53:33.341939 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.341920    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerDied","Data":"2f47feba310b021b52c4b554bae20902bb09d315edb5fcbe6b3d23a5f9b950d5"}
Jul 09 18:53:33.357930 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.357895    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9qhwv" podStartSLOduration=4.012083388 podStartE2EDuration="23.357881757s" totalImagesPullingTime="19.345798369s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:32.434108447 +0000 UTC m=+22.786543511" watchObservedRunningTime="2026-07-09 18:53:33.357881757 +0000 UTC m=+23.710316822"
Jul 09 18:53:33.397231 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.397182    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-4xkfn" podStartSLOduration=4.023608638 podStartE2EDuration="23.39716626s" totalImagesPullingTime="19.373557622s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:33.358099673 +0000 UTC m=+23.710534738" watchObservedRunningTime="2026-07-09 18:53:33.39716626 +0000 UTC m=+23.749601324"
Jul 09 18:53:33.959724 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:33.959689    2556 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 09 18:53:34.046591 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.046316    2556 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-09T18:53:33.959711615Z","UUID":"89396ac2-966c-45c2-ba49-ca97ea6f1e6a","Handler":null,"Name":"","Endpoint":""}
Jul 09 18:53:34.047902 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.047879    2556 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 09 18:53:34.047999 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.047919    2556 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 09 18:53:34.081167 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.081142    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:34.081259 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.081171    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:34.081296 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:34.081251    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:34.081330 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.081301    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:34.081393 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:34.081375    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:34.081482 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:34.081464    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:34.345603 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.345573    2556 generic.go:356] "Generic (PLEG): container finished" podID="fca2b0fb057877bc56917835668949e5" containerID="e4678d4095847d20d08e75ae521c41799793a1a65b64b861b7ac7f92f5c4b10f" exitCode=0
Jul 09 18:53:34.345991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.345644    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal" event={"ID":"fca2b0fb057877bc56917835668949e5","Type":"ContainerDied","Data":"e4678d4095847d20d08e75ae521c41799793a1a65b64b861b7ac7f92f5c4b10f"}
Jul 09 18:53:34.347435 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.347408    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r" event={"ID":"d8946aaf-6587-4254-8c07-bb2b12cf87d0","Type":"ContainerStarted","Data":"6f81ea3cb71b26515b055cc719f3d98662ab356acdc2d83a80582877e8565911"}
Jul 09 18:53:34.349951 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.349901    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 18:53:34.350303 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.350286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"b05b122d48c8c655ce03d8f0d3e3913034c2951f80edaf5896437b5bf5308b46"}
Jul 09 18:53:34.350375 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.350343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"3eeba2756a75c91fb8b6b61597202a36605ac45dcba094ed7d5ac93ad9e5a7a5"}
Jul 09 18:53:34.350375 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.350373    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"1aa3000e01a0208b59eee9bbfb2b6861545bd388f15f2826f3826d786b116529"}
Jul 09 18:53:34.350447 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.350400    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"183921951cd1187f87e9d53bd8d038d3171baa24636be95157062844c87b4466"}
Jul 09 18:53:34.669696 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.669597    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:34.670267 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:34.670242    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:36.082234 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:36.082054    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:36.082775 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:36.082091    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:36.082775 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:36.082323    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:36.082775 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:36.082129    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:36.082775 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:36.082389    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:36.082775 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:36.082433    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:36.354754 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:36.354725    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal" event={"ID":"fca2b0fb057877bc56917835668949e5","Type":"ContainerStarted","Data":"a9efb3cd8b6895c1da52577baa6b53fab2eb1b396084634cde32746f38c31507"}
Jul 09 18:53:36.356447 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:36.356425    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r" event={"ID":"d8946aaf-6587-4254-8c07-bb2b12cf87d0","Type":"ContainerStarted","Data":"93e469b2d1fcf114595e2e5fd6a0788f7e7651c8b55483a9997874759bb80428"}
Jul 09 18:53:36.428037 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:36.427990    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-93.ec2.internal" podStartSLOduration=25.427976455 podStartE2EDuration="25.427976455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:36.42755568 +0000 UTC m=+26.779990745" watchObservedRunningTime="2026-07-09 18:53:36.427976455 +0000 UTC m=+26.780411520"
Jul 09 18:53:37.361359 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:37.361334    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 18:53:37.361819 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:37.361630    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"052806f4d894cd11d27c17f0add75638d47f7696add801011c1bb3c872f09aeb"}
Jul 09 18:53:38.081926 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:38.081897    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:38.082105 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:38.081987    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:38.082105 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:38.081993    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:38.082230 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:38.082096    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:38.082230 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:38.082150    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:38.082313 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:38.082239    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:39.374236 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:39.374012    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 18:53:39.374979 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:39.374658    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"7af8dfdd66668644de305a356127c15617b05b55c4d922d946c4e574dd2d6e08"}
Jul 09 18:53:39.374979 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:39.374921    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:39.375155 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:39.375139    2556 scope.go:117] "RemoveContainer" containerID="2f47feba310b021b52c4b554bae20902bb09d315edb5fcbe6b3d23a5f9b950d5"
Jul 09 18:53:39.391672 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:39.391646    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:39.404521 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:39.404475    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-twb8r" podStartSLOduration=7.258405275 podStartE2EDuration="29.404456389s" totalImagesPullingTime="22.146051114s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:36.448338738 +0000 UTC m=+26.800773803" watchObservedRunningTime="2026-07-09 18:53:39.404456389 +0000 UTC m=+29.756891460"
Jul 09 18:53:40.082602 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.082562    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:40.082790 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:40.082704    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:40.082790 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.082756    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:40.082912 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.082748    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:40.082966 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:40.082931    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:40.083044 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:40.083022    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:40.827601 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.827567    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-66mk7"]
Jul 09 18:53:40.828005 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.827666    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:40.828005 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:40.827758    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:40.832561 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.832287    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rs466"]
Jul 09 18:53:40.832561 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.832383    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:40.832561 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:40.832481    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:40.835418 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.835393    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v9667"]
Jul 09 18:53:40.835556 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:40.835470    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:40.835615 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:40.835572    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:41.381459 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.381426    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ba3f622-358b-4925-a89d-3ac1d413b48e" containerID="ff962d4982e07fb14bae997244d6feaea026efc841fc63dd8ad082b6159e0935" exitCode=0
Jul 09 18:53:41.381609 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.381503    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerDied","Data":"ff962d4982e07fb14bae997244d6feaea026efc841fc63dd8ad082b6159e0935"}
Jul 09 18:53:41.385281 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.385258    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 18:53:41.385679 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.385660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" event={"ID":"5f6e8995-b752-4679-887f-4cd3a9826b7c","Type":"ContainerStarted","Data":"ebdf4cf1af1766a1f368caf4800548dbfa723f56d022fa80f008287f31186aea"}
Jul 09 18:53:41.385907 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.385890    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:41.385995 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.385958    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:41.400841 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.400815    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:53:41.436585 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:41.436509    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" podStartSLOduration=9.981125974 podStartE2EDuration="31.43649016s" totalImagesPullingTime="21.455364186s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:41.435742372 +0000 UTC m=+31.788177437" watchObservedRunningTime="2026-07-09 18:53:41.43649016 +0000 UTC m=+31.788925220"
Jul 09 18:53:42.078336 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:42.078298    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:42.078738 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:42.078414    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:42.078738 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:42.078444    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:42.078738 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:42.078501    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:42.095899 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:42.095866    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:42.096019 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:42.095993    2556 prober_manager.go:312] "Failed to trigger a manual run" probe="Readiness"
Jul 09 18:53:42.096428 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:42.096403    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-9gvq9"
Jul 09 18:53:43.078500 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:43.078357    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:43.078884 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.078617    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:43.391767 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:43.391732    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ba3f622-358b-4925-a89d-3ac1d413b48e" containerID="80d385bec3fff80b2e6d03b6a2f716dd104c59f309d6681eb46970dffea880e6" exitCode=0
Jul 09 18:53:43.391952 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:43.391802    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerDied","Data":"80d385bec3fff80b2e6d03b6a2f716dd104c59f309d6681eb46970dffea880e6"}
Jul 09 18:53:43.404573 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:43.404520    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5" podUID="5f6e8995-b752-4679-887f-4cd3a9826b7c" containerName="ovnkube-controller" probeResult="failure" output=""
Jul 09 18:53:43.739287 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:43.739251    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:43.739468 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.739369    2556 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:43.739468 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.739419    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret podName:f721565e-88df-4cad-b2e4-49193dcfbddd nodeName:}" failed. No retries permitted until 2026-07-09 18:54:15.739406337 +0000 UTC m=+66.091841380 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret") pod "global-pull-secret-syncer-v9667" (UID: "f721565e-88df-4cad-b2e4-49193dcfbddd") : object "kube-system"/"original-pull-secret" not registered
Jul 09 18:53:43.839774 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:43.839738    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:43.839916 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.839860    2556 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:43.839916 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.839913    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs podName:a8a09070-edcc-43d2-8b08-a3fd1950f284 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:15.839899167 +0000 UTC m=+66.192334211 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs") pod "network-metrics-daemon-66mk7" (UID: "a8a09070-edcc-43d2-8b08-a3fd1950f284") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 09 18:53:43.940660 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:43.940624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:43.940837 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.940805    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 09 18:53:43.940837 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.940826    2556 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 09 18:53:43.940837 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.940836    2556 projected.go:196] Error preparing data for projected volume kube-api-access-hzxl9 for pod openshift-network-diagnostics/network-check-target-rs466: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:43.940972 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:43.940893    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9 podName:e99ad357-fdc9-4571-9528-e5743907647a nodeName:}" failed. No retries permitted until 2026-07-09 18:54:15.940874625 +0000 UTC m=+66.293309691 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-hzxl9" (UniqueName: "kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9") pod "network-check-target-rs466" (UID: "e99ad357-fdc9-4571-9528-e5743907647a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 09 18:53:44.078868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:44.078785    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:44.079367 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:44.078785    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:44.079367 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:44.078914    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:44.079367 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:44.079020    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:45.079278 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:45.079191    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:45.079747 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:45.079316    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-v9667" podUID="f721565e-88df-4cad-b2e4-49193dcfbddd"
Jul 09 18:53:45.398265 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:45.398180    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ba3f622-358b-4925-a89d-3ac1d413b48e" containerID="d1302f270ba4215e9325d572e1b5ff1b9a3d7a3365fbf9cadbccc2708ad46a26" exitCode=0
Jul 09 18:53:45.398265 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:45.398226    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerDied","Data":"d1302f270ba4215e9325d572e1b5ff1b9a3d7a3365fbf9cadbccc2708ad46a26"}
Jul 09 18:53:46.079206 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.079160    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:46.079206 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.079212    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:46.079752 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.079289    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-rs466" podUID="e99ad357-fdc9-4571-9528-e5743907647a"
Jul 09 18:53:46.079752 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.079423    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-66mk7" podUID="a8a09070-edcc-43d2-8b08-a3fd1950f284"
Jul 09 18:53:46.501432 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.501398    2556 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-93.ec2.internal" event="NodeReady"
Jul 09 18:53:46.501643 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.501541    2556 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 09 18:53:46.543165 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.543133    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"]
Jul 09 18:53:46.558289 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.558258    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"]
Jul 09 18:53:46.558439 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.558374    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.568462 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.568435    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"]
Jul 09 18:53:46.568647 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.568498    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 09 18:53:46.568647 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.568577    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.568647 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.568642    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4w25g\""
Jul 09 18:53:46.568891 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.568869    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 09 18:53:46.569045 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.569029    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 09 18:53:46.572387 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.572368    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 09 18:53:46.576393 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.576376    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-strwc\""
Jul 09 18:53:46.576587 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.576570    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-tls\""
Jul 09 18:53:46.576663 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.576589    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.576663 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.576625    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.578099 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.578082    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemetry-config\""
Jul 09 18:53:46.584320 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.584302    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/router-default-66ddf58c96-f6w75"]
Jul 09 18:53:46.584427 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.584413    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:46.589727 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.589585    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cmpq6\""
Jul 09 18:53:46.589727 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.589631    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"samples-operator-tls\""
Jul 09 18:53:46.589727 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.589586    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.590472 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.590454    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-samples-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.594414 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.594396    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console-operator/console-operator-7866497494-9j4h9"]
Jul 09 18:53:46.594498 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.594477    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.598726 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.598708    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-metrics-certs-default\""
Jul 09 18:53:46.600151 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.599844    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"service-ca-bundle\""
Jul 09 18:53:46.600151 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.599853    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.600151 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.599886    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zrs78\""
Jul 09 18:53:46.600151 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.599907    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-ingress-cert\""
Jul 09 18:53:46.600151 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.599853    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.610153 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.610129    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-stats-default\""
Jul 09 18:53:46.610555 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.610509    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7zc7x"]
Jul 09 18:53:46.611717 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.611694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.615515 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.615495    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"console-operator-dockercfg-kh5vh\""
Jul 09 18:53:46.618481 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.618463    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.618780 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.618762    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"console-operator-config\""
Jul 09 18:53:46.619011 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.618847    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.620672 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.620648    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console-operator\"/\"serving-cert\""
Jul 09 18:53:46.627265 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.627241    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console-operator\"/\"trusted-ca\""
Jul 09 18:53:46.636728 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.636703    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"]
Jul 09 18:53:46.636925 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.636854    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.640359 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.640338    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-6q9h8\""
Jul 09 18:53:46.640660 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.640641    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Jul 09 18:53:46.640757 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.640664    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.640757 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.640644    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"openshift-insights-serving-cert\""
Jul 09 18:53:46.640944 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.640927    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.649228 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.649149    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Jul 09 18:53:46.652756 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.652734    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"]
Jul 09 18:53:46.652879 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.652851    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"
Jul 09 18:53:46.656262 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.656242    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.656357 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.656242    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.656357 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.656283    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-8xd7d\""
Jul 09 18:53:46.661659 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.661638    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.661750 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.661665    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.661750 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.661683    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-bound-sa-token\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.661750 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.661701    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc39ed10-3fcd-47e9-98af-2940484eb9ca-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.661899 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.661836    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlxw2\" (UniqueName: \"kubernetes.io/projected/bc39ed10-3fcd-47e9-98af-2940484eb9ca-kube-api-access-tlxw2\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.662027 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.662004    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-image-registry-private-configuration\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.662075 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.662038    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-registry-certificates\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.662075 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.662058    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0063f3df-5e0f-4576-8348-81aa821a2857-ca-trust-extracted\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.662162 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.662073    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xjqwb\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-kube-api-access-xjqwb\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.662162 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.662103    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-trusted-ca\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.662162 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.662121    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-installation-pull-secrets\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.668786 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668768    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66ddf58c96-f6w75"]
Jul 09 18:53:46.668874 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668814    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"]
Jul 09 18:53:46.668874 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668840    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7zc7x"]
Jul 09 18:53:46.668874 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668856    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"]
Jul 09 18:53:46.668874 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668773    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.668874 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668874    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"]
Jul 09 18:53:46.669084 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668888    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-9j4h9"]
Jul 09 18:53:46.669084 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668903    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"]
Jul 09 18:53:46.669084 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.668935    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-5knnr"]
Jul 09 18:53:46.672214 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.672193    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.672322 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.672263    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"serving-cert\""
Jul 09 18:53:46.672642 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.672466    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.672642 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.672487    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-dockercfg-wfglx\""
Jul 09 18:53:46.672642 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.672573    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca-operator\"/\"service-ca-operator-config\""
Jul 09 18:53:46.684845 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.684821    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"]
Jul 09 18:53:46.684954 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.684943    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.688464 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.688439    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xvss5\""
Jul 09 18:53:46.688464 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.688438    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 09 18:53:46.688898 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.688880    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 09 18:53:46.700966 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.700938    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-ltkwk"]
Jul 09 18:53:46.701102 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.701083    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:46.706991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.706970    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Jul 09 18:53:46.706991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.706986    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Jul 09 18:53:46.707288 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.707232    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-szg6x\""
Jul 09 18:53:46.724503 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.723273    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"]
Jul 09 18:53:46.724841 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.724820    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:46.727330 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.727304    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:46.729713 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.729694    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.729813 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.729735    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 09 18:53:46.730025 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.730004    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-tl7rk\""
Jul 09 18:53:46.731747 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.731727    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.743291 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.743266    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"]
Jul 09 18:53:46.743396 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.743380    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"
Jul 09 18:53:46.746514 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.746496    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.747842 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.747826    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-r2nkq\""
Jul 09 18:53:46.748121 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.748105    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.762594 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762558    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsphh\" (UniqueName: \"kubernetes.io/projected/92ee9458-91f1-4c04-8144-29691b182b9d-kube-api-access-dsphh\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:46.762713 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762605    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/569b9f3d-0fa3-425a-9815-a374ea2780a6-tmp\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.762713 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762631    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569b9f3d-0fa3-425a-9815-a374ea2780a6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.762713 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762665    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-image-registry-private-configuration\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.762713 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762687    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.762713 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762707    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0063f3df-5e0f-4576-8348-81aa821a2857-ca-trust-extracted\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.762963 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-trusted-ca\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.762963 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/569b9f3d-0fa3-425a-9815-a374ea2780a6-serving-cert\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.762963 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762806    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:46.762963 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762840    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v7mbb\" (UniqueName: \"kubernetes.io/projected/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-kube-api-access-v7mbb\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.762963 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.762963 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.762894    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.763237 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.763018    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:46.763237 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.763029    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:53:46.763237 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.763046    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4b4f9d7-n2cgw: secret "image-registry-tls" not found
Jul 09 18:53:46.763237 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763077    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0063f3df-5e0f-4576-8348-81aa821a2857-ca-trust-extracted\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.763237 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.763106    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls podName:bc39ed10-3fcd-47e9-98af-2940484eb9ca nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.263087227 +0000 UTC m=+37.615522270 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-rbkwz" (UID: "bc39ed10-3fcd-47e9-98af-2940484eb9ca") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:46.763237 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.763132    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls podName:0063f3df-5e0f-4576-8348-81aa821a2857 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.263115818 +0000 UTC m=+37.615550862 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls") pod "image-registry-7c4b4f9d7-n2cgw" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857") : secret "image-registry-tls" not found
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-bound-sa-token\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763331    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763354    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc39ed10-3fcd-47e9-98af-2940484eb9ca-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763369    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/569b9f3d-0fa3-425a-9815-a374ea2780a6-snapshots\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763389    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-default-certificate\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763434    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-stats-auth\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763488    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x2qbf\" (UniqueName: \"kubernetes.io/projected/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-kube-api-access-x2qbf\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.763540 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763513    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzd2f\" (UniqueName: \"kubernetes.io/projected/569b9f3d-0fa3-425a-9815-a374ea2780a6-kube-api-access-mzd2f\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763561    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlxw2\" (UniqueName: \"kubernetes.io/projected/bc39ed10-3fcd-47e9-98af-2940484eb9ca-kube-api-access-tlxw2\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763586    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-registry-certificates\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763626    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xjqwb\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-kube-api-access-xjqwb\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763657    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-serving-cert\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763680    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q98v7\" (UniqueName: \"kubernetes.io/projected/ca773c32-ad49-44eb-855d-64f51ae05dfa-kube-api-access-q98v7\") pod \"volume-data-source-validator-78495597f6-lh884\" (UID: \"ca773c32-ad49-44eb-855d-64f51ae05dfa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763712    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-installation-pull-secrets\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763738    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569b9f3d-0fa3-425a-9815-a374ea2780a6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763772    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cprq7\" (UniqueName: \"kubernetes.io/projected/7d5933ef-2952-4201-891b-78f29352a829-kube-api-access-cprq7\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763800    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-trusted-ca\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.763897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763827    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.764275 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763943    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-config\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.764275 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.763977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-config\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.764275 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.764003    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-trusted-ca\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.764275 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.764067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemetry-config\" (UniqueName: \"kubernetes.io/configmap/bc39ed10-3fcd-47e9-98af-2940484eb9ca-telemetry-config\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.764275 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.764124    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-registry-certificates\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.767195 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.767173    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-image-registry-private-configuration\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.767287 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.767179    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-installation-pull-secrets\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.769511 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.769493    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"]
Jul 09 18:53:46.769584 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.769563    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5knnr"]
Jul 09 18:53:46.769618 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.769591    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ltkwk"]
Jul 09 18:53:46.769618 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.769611    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"]
Jul 09 18:53:46.769676 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.769615    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.769676 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.769629    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"]
Jul 09 18:53:46.769676 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.769648    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"]
Jul 09 18:53:46.773218 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.773202    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"serving-cert\""
Jul 09 18:53:46.773499 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.773480    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-root-ca.crt\""
Jul 09 18:53:46.773695 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.773682    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:46.774063 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.774048    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"config\""
Jul 09 18:53:46.774262 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.774247    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator-operator\"/\"kube-storage-version-migrator-operator-dockercfg-lt758\""
Jul 09 18:53:46.777601 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.777577    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlxw2\" (UniqueName: \"kubernetes.io/projected/bc39ed10-3fcd-47e9-98af-2940484eb9ca-kube-api-access-tlxw2\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:46.777885 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.777867    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-bound-sa-token\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.777947 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.777869    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xjqwb\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-kube-api-access-xjqwb\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:46.860313 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.860276    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:46.865249 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865225    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n6qfg\" (UniqueName: \"kubernetes.io/projected/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-kube-api-access-n6qfg\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.865378 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865256    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/569b9f3d-0fa3-425a-9815-a374ea2780a6-serving-cert\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.865378 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865274    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:46.865378 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865331    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v7mbb\" (UniqueName: \"kubernetes.io/projected/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-kube-api-access-v7mbb\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.865378 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865355    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2000c44b-544b-4072-8d0a-604d4b1e1d47-tmp-dir\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.865597 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.865441    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:53:46.865597 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865469    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rlmf7\" (UniqueName: \"kubernetes.io/projected/2000c44b-544b-4072-8d0a-604d4b1e1d47-kube-api-access-rlmf7\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.865597 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.865511    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls podName:92ee9458-91f1-4c04-8144-29691b182b9d nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.365489678 +0000 UTC m=+37.717924741 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls") pod "cluster-samples-operator-65db858894-zjtpg" (UID: "92ee9458-91f1-4c04-8144-29691b182b9d") : secret "samples-operator-tls" not found
Jul 09 18:53:46.865597 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865573    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865604    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/569b9f3d-0fa3-425a-9815-a374ea2780a6-snapshots\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865632    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865659    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-default-certificate\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.865694    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865713    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-stats-auth\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.865743    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.365730053 +0000 UTC m=+37.718165096 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : secret "router-metrics-certs-default" not found
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865775    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x2qbf\" (UniqueName: \"kubernetes.io/projected/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-kube-api-access-x2qbf\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.865794 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865795    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzd2f\" (UniqueName: \"kubernetes.io/projected/569b9f3d-0fa3-425a-9815-a374ea2780a6-kube-api-access-mzd2f\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865818    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865847    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2000c44b-544b-4072-8d0a-604d4b1e1d47-config-volume\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865875    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b49c8ad2-d3d4-44bd-91d1-d454284d0c64-cert\") pod \"ingress-canary-ltkwk\" (UID: \"b49c8ad2-d3d4-44bd-91d1-d454284d0c64\") " pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865902    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-serving-cert\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865928    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q98v7\" (UniqueName: \"kubernetes.io/projected/ca773c32-ad49-44eb-855d-64f51ae05dfa-kube-api-access-q98v7\") pod \"volume-data-source-validator-78495597f6-lh884\" (UID: \"ca773c32-ad49-44eb-855d-64f51ae05dfa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865954    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569b9f3d-0fa3-425a-9815-a374ea2780a6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.865977    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2nn2\" (UniqueName: \"kubernetes.io/projected/b716f137-6c6d-4f46-abe9-4c011ac3fc63-kube-api-access-l2nn2\") pod \"network-check-source-869dcfcdcd-ncr7h\" (UID: \"b716f137-6c6d-4f46-abe9-4c011ac3fc63\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"
Jul 09 18:53:46.866189 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866169    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"snapshots\" (UniqueName: \"kubernetes.io/empty-dir/569b9f3d-0fa3-425a-9815-a374ea2780a6-snapshots\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0208420e-d48a-418d-b036-9a1a611ad6bf-nginx-conf\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866316    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cprq7\" (UniqueName: \"kubernetes.io/projected/7d5933ef-2952-4201-891b-78f29352a829-kube-api-access-cprq7\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866382    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-trusted-ca\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866416    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sjsf\" (UniqueName: \"kubernetes.io/projected/b49c8ad2-d3d4-44bd-91d1-d454284d0c64-kube-api-access-8sjsf\") pod \"ingress-canary-ltkwk\" (UID: \"b49c8ad2-d3d4-44bd-91d1-d454284d0c64\") " pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866449    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866488    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-config\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-config\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866559    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsphh\" (UniqueName: \"kubernetes.io/projected/92ee9458-91f1-4c04-8144-29691b182b9d-kube-api-access-dsphh\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:46.866611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866600    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/569b9f3d-0fa3-425a-9815-a374ea2780a6-tmp\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.867081 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.866626    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.366609805 +0000 UTC m=+37.719044848 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : configmap references non-existent config key: service-ca.crt
Jul 09 18:53:46.867081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866640    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569b9f3d-0fa3-425a-9815-a374ea2780a6-service-ca-bundle\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.867081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866661    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.867081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866696    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569b9f3d-0fa3-425a-9815-a374ea2780a6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.867081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866729    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.867081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866758    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.867081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.866841    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/569b9f3d-0fa3-425a-9815-a374ea2780a6-tmp\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.867607 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.867308    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-trusted-ca\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.867607 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.867485    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-config\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.868056 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.868014    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/569b9f3d-0fa3-425a-9815-a374ea2780a6-trusted-ca-bundle\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.868163 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.868098    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-config\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.868382 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.868361    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"stats-auth\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-stats-auth\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.868491 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.868467    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"default-certificate\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-default-certificate\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.868789 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.868751    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/569b9f3d-0fa3-425a-9815-a374ea2780a6-serving-cert\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.869226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.869210    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-serving-cert\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.869651 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.869634    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-serving-cert\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.878109 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.878090    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v7mbb\" (UniqueName: \"kubernetes.io/projected/bfe48b74-fb4f-4e95-b58d-c05fb602c43c-kube-api-access-v7mbb\") pod \"service-ca-operator-7695fc6c8c-qmxxz\" (UID: \"bfe48b74-fb4f-4e95-b58d-c05fb602c43c\") " pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.880832 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.880807    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cprq7\" (UniqueName: \"kubernetes.io/projected/7d5933ef-2952-4201-891b-78f29352a829-kube-api-access-cprq7\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:46.881042 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.881025    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q98v7\" (UniqueName: \"kubernetes.io/projected/ca773c32-ad49-44eb-855d-64f51ae05dfa-kube-api-access-q98v7\") pod \"volume-data-source-validator-78495597f6-lh884\" (UID: \"ca773c32-ad49-44eb-855d-64f51ae05dfa\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"
Jul 09 18:53:46.881262 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.881244    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x2qbf\" (UniqueName: \"kubernetes.io/projected/6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3-kube-api-access-x2qbf\") pod \"console-operator-7866497494-9j4h9\" (UID: \"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3\") " pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.882002 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.881985    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzd2f\" (UniqueName: \"kubernetes.io/projected/569b9f3d-0fa3-425a-9815-a374ea2780a6-kube-api-access-mzd2f\") pod \"insights-operator-68d9dd94cf-7zc7x\" (UID: \"569b9f3d-0fa3-425a-9815-a374ea2780a6\") " pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.886101 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.886082    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsphh\" (UniqueName: \"kubernetes.io/projected/92ee9458-91f1-4c04-8144-29691b182b9d-kube-api-access-dsphh\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:46.924867 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.924840    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:53:46.947174 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.947151    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x"
Jul 09 18:53:46.964448 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.964422    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"
Jul 09 18:53:46.967792 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967768    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n6qfg\" (UniqueName: \"kubernetes.io/projected/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-kube-api-access-n6qfg\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.967929 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2000c44b-544b-4072-8d0a-604d4b1e1d47-tmp-dir\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.967929 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rlmf7\" (UniqueName: \"kubernetes.io/projected/2000c44b-544b-4072-8d0a-604d4b1e1d47-kube-api-access-rlmf7\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.967929 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967885    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.967929 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967909    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:46.967929 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967928    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2000c44b-544b-4072-8d0a-604d4b1e1d47-config-volume\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.968175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967959    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b49c8ad2-d3d4-44bd-91d1-d454284d0c64-cert\") pod \"ingress-canary-ltkwk\" (UID: \"b49c8ad2-d3d4-44bd-91d1-d454284d0c64\") " pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:46.968175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.967984    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2nn2\" (UniqueName: \"kubernetes.io/projected/b716f137-6c6d-4f46-abe9-4c011ac3fc63-kube-api-access-l2nn2\") pod \"network-check-source-869dcfcdcd-ncr7h\" (UID: \"b716f137-6c6d-4f46-abe9-4c011ac3fc63\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"
Jul 09 18:53:46.968175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.968004    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0208420e-d48a-418d-b036-9a1a611ad6bf-nginx-conf\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:46.968175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.968029    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sjsf\" (UniqueName: \"kubernetes.io/projected/b49c8ad2-d3d4-44bd-91d1-d454284d0c64-kube-api-access-8sjsf\") pod \"ingress-canary-ltkwk\" (UID: \"b49c8ad2-d3d4-44bd-91d1-d454284d0c64\") " pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:46.968175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.968064    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.968175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.968082    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.968175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.968114    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2000c44b-544b-4072-8d0a-604d4b1e1d47-tmp-dir\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.968669 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.968208    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:53:46.968669 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.968262    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls podName:2000c44b-544b-4072-8d0a-604d4b1e1d47 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.468243916 +0000 UTC m=+37.820678966 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls") pod "dns-default-5knnr" (UID: "2000c44b-544b-4072-8d0a-604d4b1e1d47") : secret "dns-default-metrics-tls" not found
Jul 09 18:53:46.968669 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.968366    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:53:46.968669 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:46.968412    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert podName:0208420e-d48a-418d-b036-9a1a611ad6bf nodeName:}" failed. No retries permitted until 2026-07-09 18:53:47.468399053 +0000 UTC m=+37.820834095 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t2zpb" (UID: "0208420e-d48a-418d-b036-9a1a611ad6bf") : secret "networking-console-plugin-cert" not found
Jul 09 18:53:46.968669 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.968590    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/2000c44b-544b-4072-8d0a-604d4b1e1d47-config-volume\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.969082 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.969039    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/configmap/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-config\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.969172 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.969087    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/0208420e-d48a-418d-b036-9a1a611ad6bf-nginx-conf\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:46.970852 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.970832    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b49c8ad2-d3d4-44bd-91d1-d454284d0c64-cert\") pod \"ingress-canary-ltkwk\" (UID: \"b49c8ad2-d3d4-44bd-91d1-d454284d0c64\") " pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:46.971136 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.971113    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-cert\" (UniqueName: \"kubernetes.io/secret/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-serving-cert\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.981237 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.980967    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"
Jul 09 18:53:46.986938 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.986888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n6qfg\" (UniqueName: \"kubernetes.io/projected/ff7c7467-a4b1-44aa-8dd8-50ed702d7f87-kube-api-access-n6qfg\") pod \"kube-storage-version-migrator-operator-799cb7f958-rgrvz\" (UID: \"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87\") " pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:46.991026 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.990937    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rlmf7\" (UniqueName: \"kubernetes.io/projected/2000c44b-544b-4072-8d0a-604d4b1e1d47-kube-api-access-rlmf7\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:46.991184 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.991067    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sjsf\" (UniqueName: \"kubernetes.io/projected/b49c8ad2-d3d4-44bd-91d1-d454284d0c64-kube-api-access-8sjsf\") pod \"ingress-canary-ltkwk\" (UID: \"b49c8ad2-d3d4-44bd-91d1-d454284d0c64\") " pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:46.991630 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:46.991607    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2nn2\" (UniqueName: \"kubernetes.io/projected/b716f137-6c6d-4f46-abe9-4c011ac3fc63-kube-api-access-l2nn2\") pod \"network-check-source-869dcfcdcd-ncr7h\" (UID: \"b716f137-6c6d-4f46-abe9-4c011ac3fc63\") " pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"
Jul 09 18:53:47.037767 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.037352    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-ltkwk"
Jul 09 18:53:47.052827 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.052749    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"
Jul 09 18:53:47.081655 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.080892    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:53:47.085800 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.084176    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:53:47.092567 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.090725    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"
Jul 09 18:53:47.207111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.204954    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884"]
Jul 09 18:53:47.207111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.205949    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console-operator/console-operator-7866497494-9j4h9"]
Jul 09 18:53:47.210921 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.210728    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-operator-68d9dd94cf-7zc7x"]
Jul 09 18:53:47.212074 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:47.212044    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca773c32_ad49_44eb_855d_64f51ae05dfa.slice/crio-687677a0b4870300fd3aa154567f5d90439d58172074999b9b215a0f835de035 WatchSource:0}: Error finding container 687677a0b4870300fd3aa154567f5d90439d58172074999b9b215a0f835de035: Status 404 returned error can't find the container with id 687677a0b4870300fd3aa154567f5d90439d58172074999b9b215a0f835de035
Jul 09 18:53:47.214612 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:47.214574    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fd1740a_a8b9_4cdd_87be_0fa546ddf5e3.slice/crio-7fef024f7ff09273b40103ff149097cfe9d4392aca1fb5b74977b71d465474da WatchSource:0}: Error finding container 7fef024f7ff09273b40103ff149097cfe9d4392aca1fb5b74977b71d465474da: Status 404 returned error can't find the container with id 7fef024f7ff09273b40103ff149097cfe9d4392aca1fb5b74977b71d465474da
Jul 09 18:53:47.230919 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.230135    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz"]
Jul 09 18:53:47.234431 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:47.234404    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbfe48b74_fb4f_4e95_b58d_c05fb602c43c.slice/crio-7aab16a9e099647e83cc3e1af892546a64d22c94f002a878c55cae994a073d4f WatchSource:0}: Error finding container 7aab16a9e099647e83cc3e1af892546a64d22c94f002a878c55cae994a073d4f: Status 404 returned error can't find the container with id 7aab16a9e099647e83cc3e1af892546a64d22c94f002a878c55cae994a073d4f
Jul 09 18:53:47.272976 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.272940    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:47.272976 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.272986    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:47.273231 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.273100    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:47.273231 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.273126    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:53:47.273231 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.273146    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4b4f9d7-n2cgw: secret "image-registry-tls" not found
Jul 09 18:53:47.273231 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.273180    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls podName:bc39ed10-3fcd-47e9-98af-2940484eb9ca nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.273158463 +0000 UTC m=+38.625593509 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-rbkwz" (UID: "bc39ed10-3fcd-47e9-98af-2940484eb9ca") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:47.273231 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.273201    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls podName:0063f3df-5e0f-4576-8348-81aa821a2857 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.273190866 +0000 UTC m=+38.625625908 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls") pod "image-registry-7c4b4f9d7-n2cgw" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857") : secret "image-registry-tls" not found
Jul 09 18:53:47.287067 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.285851    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-ltkwk"]
Jul 09 18:53:47.305926 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.305903    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz"]
Jul 09 18:53:47.308802 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:47.308776    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff7c7467_a4b1_44aa_8dd8_50ed702d7f87.slice/crio-d53ee88e00d9a0cf5e37a57261464e3bdd60744710e952919029e0bea7b41e73 WatchSource:0}: Error finding container d53ee88e00d9a0cf5e37a57261464e3bdd60744710e952919029e0bea7b41e73: Status 404 returned error can't find the container with id d53ee88e00d9a0cf5e37a57261464e3bdd60744710e952919029e0bea7b41e73
Jul 09 18:53:47.312070 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.312044    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h"]
Jul 09 18:53:47.315558 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:47.315519    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb716f137_6c6d_4f46_abe9_4c011ac3fc63.slice/crio-15d0e06b233e405d5467cee708af77b9e622d574ee343b390df435e21aee7df2 WatchSource:0}: Error finding container 15d0e06b233e405d5467cee708af77b9e622d574ee343b390df435e21aee7df2: Status 404 returned error can't find the container with id 15d0e06b233e405d5467cee708af77b9e622d574ee343b390df435e21aee7df2
Jul 09 18:53:47.373843 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.373800    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:47.374027 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.373896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:47.374027 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.373966    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:53:47.374027 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.373998    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:47.374176 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.374057    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls podName:92ee9458-91f1-4c04-8144-29691b182b9d nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.374026814 +0000 UTC m=+38.726461874 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls") pod "cluster-samples-operator-65db858894-zjtpg" (UID: "92ee9458-91f1-4c04-8144-29691b182b9d") : secret "samples-operator-tls" not found
Jul 09 18:53:47.374176 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.374056    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:53:47.374176 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.374125    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.374105402 +0000 UTC m=+38.726540463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : configmap references non-existent config key: service-ca.crt
Jul 09 18:53:47.374176 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.374153    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.374141964 +0000 UTC m=+38.726577021 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : secret "router-metrics-certs-default" not found
Jul 09 18:53:47.403729 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.403697    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-9j4h9" event={"ID":"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3","Type":"ContainerStarted","Data":"7fef024f7ff09273b40103ff149097cfe9d4392aca1fb5b74977b71d465474da"}
Jul 09 18:53:47.405028 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.404999    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884" event={"ID":"ca773c32-ad49-44eb-855d-64f51ae05dfa","Type":"ContainerStarted","Data":"687677a0b4870300fd3aa154567f5d90439d58172074999b9b215a0f835de035"}
Jul 09 18:53:47.406419 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.406393    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x" event={"ID":"569b9f3d-0fa3-425a-9815-a374ea2780a6","Type":"ContainerStarted","Data":"7c9c8721c9c26db8700c7b685b8a3b73cdd8b5d9bb05314f990b365d85c3d36f"}
Jul 09 18:53:47.408094 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.408073    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz" event={"ID":"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87","Type":"ContainerStarted","Data":"d53ee88e00d9a0cf5e37a57261464e3bdd60744710e952919029e0bea7b41e73"}
Jul 09 18:53:47.409196 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.409177    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h" event={"ID":"b716f137-6c6d-4f46-abe9-4c011ac3fc63","Type":"ContainerStarted","Data":"15d0e06b233e405d5467cee708af77b9e622d574ee343b390df435e21aee7df2"}
Jul 09 18:53:47.410169 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.410150    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ltkwk" event={"ID":"b49c8ad2-d3d4-44bd-91d1-d454284d0c64","Type":"ContainerStarted","Data":"9240f3e1e8827942bdb1ccf4c6c2ec7fa992a5c2552b827c634d049504e91e7c"}
Jul 09 18:53:47.411158 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.411137    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz" event={"ID":"bfe48b74-fb4f-4e95-b58d-c05fb602c43c","Type":"ContainerStarted","Data":"7aab16a9e099647e83cc3e1af892546a64d22c94f002a878c55cae994a073d4f"}
Jul 09 18:53:47.475057 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.474972    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:47.475057 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:47.475025    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:47.475259 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.475150    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:53:47.475259 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.475200    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:53:47.475259 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.475230    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls podName:2000c44b-544b-4072-8d0a-604d4b1e1d47 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.475207966 +0000 UTC m=+38.827643014 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls") pod "dns-default-5knnr" (UID: "2000c44b-544b-4072-8d0a-604d4b1e1d47") : secret "dns-default-metrics-tls" not found
Jul 09 18:53:47.475259 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:47.475253    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert podName:0208420e-d48a-418d-b036-9a1a611ad6bf nodeName:}" failed. No retries permitted until 2026-07-09 18:53:48.47523673 +0000 UTC m=+38.827671773 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t2zpb" (UID: "0208420e-d48a-418d-b036-9a1a611ad6bf") : secret "networking-console-plugin-cert" not found
Jul 09 18:53:48.078864 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.078825    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:53:48.079360 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.079343    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:53:48.084931 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.084907    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkpmb\""
Jul 09 18:53:48.085316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.085166    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:53:48.085372 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.085355    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47bt6\""
Jul 09 18:53:48.282600 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.281868    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:48.282600 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.281917    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:48.282600 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.282019    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:48.282600 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.282094    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls podName:bc39ed10-3fcd-47e9-98af-2940484eb9ca nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.282075197 +0000 UTC m=+40.634510258 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-rbkwz" (UID: "bc39ed10-3fcd-47e9-98af-2940484eb9ca") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:48.282600 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.282492    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:53:48.282600 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.282507    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4b4f9d7-n2cgw: secret "image-registry-tls" not found
Jul 09 18:53:48.282600 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.282580    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls podName:0063f3df-5e0f-4576-8348-81aa821a2857 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.282556832 +0000 UTC m=+40.634991875 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls") pod "image-registry-7c4b4f9d7-n2cgw" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857") : secret "image-registry-tls" not found
Jul 09 18:53:48.383051 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.382970    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:48.383204 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.383067    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:48.383204 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.383165    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:48.384184 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.383362    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.383343066 +0000 UTC m=+40.735778120 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : configmap references non-existent config key: service-ca.crt
Jul 09 18:53:48.384184 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.383878    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:53:48.384184 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.383931    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls podName:92ee9458-91f1-4c04-8144-29691b182b9d nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.383915491 +0000 UTC m=+40.736350537 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls") pod "cluster-samples-operator-65db858894-zjtpg" (UID: "92ee9458-91f1-4c04-8144-29691b182b9d") : secret "samples-operator-tls" not found
Jul 09 18:53:48.384184 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.383995    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:53:48.384184 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.384029    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.384018017 +0000 UTC m=+40.736453063 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : secret "router-metrics-certs-default" not found
Jul 09 18:53:48.484820 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.483877    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:48.484820 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:48.483932    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:48.484820 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.484130    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:53:48.484820 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.484208    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert podName:0208420e-d48a-418d-b036-9a1a611ad6bf nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.484189102 +0000 UTC m=+40.836624150 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t2zpb" (UID: "0208420e-d48a-418d-b036-9a1a611ad6bf") : secret "networking-console-plugin-cert" not found
Jul 09 18:53:48.484820 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.484648    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:53:48.484820 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:48.484696    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls podName:2000c44b-544b-4072-8d0a-604d4b1e1d47 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:50.484681239 +0000 UTC m=+40.837116288 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls") pod "dns-default-5knnr" (UID: "2000c44b-544b-4072-8d0a-604d4b1e1d47") : secret "dns-default-metrics-tls" not found
Jul 09 18:53:49.957075 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:49.957041    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:50.101787 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.101757    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:50.304591 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.304437    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:50.304591 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.304491    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:50.304869 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.304774    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:53:50.304869 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.304790    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4b4f9d7-n2cgw: secret "image-registry-tls" not found
Jul 09 18:53:50.304869 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.304848    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls podName:0063f3df-5e0f-4576-8348-81aa821a2857 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.304830236 +0000 UTC m=+44.657265283 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls") pod "image-registry-7c4b4f9d7-n2cgw" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857") : secret "image-registry-tls" not found
Jul 09 18:53:50.305203 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.305150    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:50.305203 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.305205    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls podName:bc39ed10-3fcd-47e9-98af-2940484eb9ca nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.305189585 +0000 UTC m=+44.657624634 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-rbkwz" (UID: "bc39ed10-3fcd-47e9-98af-2940484eb9ca") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:50.405494 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.405457    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:50.405672 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.405611    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:50.405672 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.405638    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:53:50.405797 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.405708    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls podName:92ee9458-91f1-4c04-8144-29691b182b9d nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.40568905 +0000 UTC m=+44.758124116 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls") pod "cluster-samples-operator-65db858894-zjtpg" (UID: "92ee9458-91f1-4c04-8144-29691b182b9d") : secret "samples-operator-tls" not found
Jul 09 18:53:50.405797 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.405772    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:53:50.405797 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.405772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:50.405945 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.405822    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.405807321 +0000 UTC m=+44.758242379 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : secret "router-metrics-certs-default" not found
Jul 09 18:53:50.405945 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.405895    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.405883458 +0000 UTC m=+44.758318504 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : configmap references non-existent config key: service-ca.crt
Jul 09 18:53:50.506335 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.506295    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:50.506541 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.506352    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:50.506621 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.506560    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:53:50.506676 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.506636    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert podName:0208420e-d48a-418d-b036-9a1a611ad6bf nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.506616518 +0000 UTC m=+44.859051561 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t2zpb" (UID: "0208420e-d48a-418d-b036-9a1a611ad6bf") : secret "networking-console-plugin-cert" not found
Jul 09 18:53:50.506741 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.506698    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:53:50.506787 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:50.506760    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls podName:2000c44b-544b-4072-8d0a-604d4b1e1d47 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:54.506741815 +0000 UTC m=+44.859176876 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls") pod "dns-default-5knnr" (UID: "2000c44b-544b-4072-8d0a-604d4b1e1d47") : secret "dns-default-metrics-tls" not found
Jul 09 18:53:50.864295 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.864265    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:50.951812 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:50.951724    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:54.340076 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:54.340034    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:53:54.340076 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:54.340079    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:53:54.340620 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.340189    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:53:54.340620 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.340204    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4b4f9d7-n2cgw: secret "image-registry-tls" not found
Jul 09 18:53:54.340620 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.340187    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:54.340620 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.340267    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls podName:0063f3df-5e0f-4576-8348-81aa821a2857 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.340247639 +0000 UTC m=+52.692682699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls") pod "image-registry-7c4b4f9d7-n2cgw" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857") : secret "image-registry-tls" not found
Jul 09 18:53:54.340620 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.340303    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls podName:bc39ed10-3fcd-47e9-98af-2940484eb9ca nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.340287779 +0000 UTC m=+52.692722821 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-rbkwz" (UID: "bc39ed10-3fcd-47e9-98af-2940484eb9ca") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:53:54.440737 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:54.440705    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:54.440926 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:54.440812    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:53:54.440926 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:54.440876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:53:54.440926 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.440882    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:53:54.441106 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.440949    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.440930093 +0000 UTC m=+52.793365149 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : secret "router-metrics-certs-default" not found
Jul 09 18:53:54.441106 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.440959    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:53:54.441106 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.440968    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.440958728 +0000 UTC m=+52.793393771 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : configmap references non-existent config key: service-ca.crt
Jul 09 18:53:54.441106 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.441000    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls podName:92ee9458-91f1-4c04-8144-29691b182b9d nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.44098535 +0000 UTC m=+52.793420410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls") pod "cluster-samples-operator-65db858894-zjtpg" (UID: "92ee9458-91f1-4c04-8144-29691b182b9d") : secret "samples-operator-tls" not found
Jul 09 18:53:54.541475 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:54.541444    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:53:54.541651 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:54.541495    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:53:54.541651 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.541625    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:53:54.541742 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.541650    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:53:54.541742 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.541705    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls podName:2000c44b-544b-4072-8d0a-604d4b1e1d47 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.541682546 +0000 UTC m=+52.894117611 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls") pod "dns-default-5knnr" (UID: "2000c44b-544b-4072-8d0a-604d4b1e1d47") : secret "dns-default-metrics-tls" not found
Jul 09 18:53:54.541742 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:54.541725    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert podName:0208420e-d48a-418d-b036-9a1a611ad6bf nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.541715059 +0000 UTC m=+52.894150108 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t2zpb" (UID: "0208420e-d48a-418d-b036-9a1a611ad6bf") : secret "networking-console-plugin-cert" not found
Jul 09 18:53:57.454830 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.454474    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x" event={"ID":"569b9f3d-0fa3-425a-9815-a374ea2780a6","Type":"ContainerStarted","Data":"ac885175eaed1f7944b6b4b5030dd68ccbcfa3af32038f00bcc4f8eb2cf63a60"}
Jul 09 18:53:57.457272 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.457203    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz" event={"ID":"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87","Type":"ContainerStarted","Data":"ccabb6dfb32860fec0bb901ba87025f5ad66341202f80aa3469a0717bd18917c"}
Jul 09 18:53:57.459116 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.459072    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-ltkwk" event={"ID":"b49c8ad2-d3d4-44bd-91d1-d454284d0c64","Type":"ContainerStarted","Data":"df0edc2e02812d68a60cd1fb651838e6f12625153038641c30d000b54a3ba239"}
Jul 09 18:53:57.460713 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.460660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz" event={"ID":"bfe48b74-fb4f-4e95-b58d-c05fb602c43c","Type":"ContainerStarted","Data":"2fe0c35ee1471966ac56d38a074549b63420fc2e233d74e60df807164322590b"}
Jul 09 18:53:57.462705 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.462685    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/0.log"
Jul 09 18:53:57.462832 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.462814    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-9j4h9" event={"ID":"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3","Type":"ContainerStarted","Data":"134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1"}
Jul 09 18:53:57.463026 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.463011    2556 scope.go:117] "RemoveContainer" containerID="134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1"
Jul 09 18:53:57.464917 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.464842    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884" event={"ID":"ca773c32-ad49-44eb-855d-64f51ae05dfa","Type":"ContainerStarted","Data":"fd17a2bd63b6478030fd7f020d2b05d30b1faeb58b8ac00613383f95dba61442"}
Jul 09 18:53:57.473924 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.473865    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x" podStartSLOduration=26.505334275 podStartE2EDuration="36.473848596s" totalImagesPullingTime="9.968514321s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:57.473153409 +0000 UTC m=+47.825588478" watchObservedRunningTime="2026-07-09 18:53:57.473848596 +0000 UTC m=+47.826283664"
Jul 09 18:53:57.511013 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.509390    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz" podStartSLOduration=26.546595379 podStartE2EDuration="36.50936931s" totalImagesPullingTime="9.962773931s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:57.509101858 +0000 UTC m=+47.861536924" watchObservedRunningTime="2026-07-09 18:53:57.50936931 +0000 UTC m=+47.861804376"
Jul 09 18:53:57.615709 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:57.615620    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:58.469583 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.469555    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 18:53:58.470026 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.469998    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/0.log"
Jul 09 18:53:58.470092 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.470039    2556 generic.go:356] "Generic (PLEG): container finished" podID="6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3" containerID="134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1" exitCode=255
Jul 09 18:53:58.470092 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.470059    2556 generic.go:356] "Generic (PLEG): container finished" podID="6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3" containerID="5d1cfaaa41fbc3d4ef0b3452655d3f46ccb7753a98f49227a7510ff652c4c06c" exitCode=255
Jul 09 18:53:58.470190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.470091    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-9j4h9" event={"ID":"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3","Type":"ContainerDied","Data":"134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1"}
Jul 09 18:53:58.470190 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.470148    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-9j4h9" event={"ID":"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3","Type":"ContainerDied","Data":"5d1cfaaa41fbc3d4ef0b3452655d3f46ccb7753a98f49227a7510ff652c4c06c"}
Jul 09 18:53:58.470267 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.470177    2556 scope.go:117] "RemoveContainer" containerID="134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1"
Jul 09 18:53:58.470378 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.470360    2556 scope.go:117] "RemoveContainer" containerID="5d1cfaaa41fbc3d4ef0b3452655d3f46ccb7753a98f49227a7510ff652c4c06c"
Jul 09 18:53:58.470627 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:58.470563    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-9j4h9_openshift-console-operator(6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3)\"" pod="openshift-console-operator/console-operator-7866497494-9j4h9" podUID="6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3"
Jul 09 18:53:58.472137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.472109    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h" event={"ID":"b716f137-6c6d-4f46-abe9-4c011ac3fc63","Type":"ContainerStarted","Data":"1e46c2a1113c350aa8acd3fc5d3d6cfb243f70f639b594d0087b62e7611ca905"}
Jul 09 18:53:58.475086 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.475064    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ba3f622-358b-4925-a89d-3ac1d413b48e" containerID="cc64980ce5d6ce59f598f19f0c5af1e30d813b1e475eab533833bd656b8a3cce" exitCode=0
Jul 09 18:53:58.475210 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.475167    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerDied","Data":"cc64980ce5d6ce59f598f19f0c5af1e30d813b1e475eab533833bd656b8a3cce"}
Jul 09 18:53:58.493783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.493734    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-lh884" podStartSLOduration=28.018363468 podStartE2EDuration="37.493716327s" totalImagesPullingTime="9.475352859s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:57.528015452 +0000 UTC m=+47.880450521" watchObservedRunningTime="2026-07-09 18:53:58.493716327 +0000 UTC m=+48.846151393"
Jul 09 18:53:58.497384 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.497322    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:58.497649 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.497631    2556 scope.go:117] "RemoveContainer" containerID="134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1"
Jul 09 18:53:58.497962 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:58.497938    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1\": container with ID starting with 134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1 not found: ID does not exist" containerID="134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1"
Jul 09 18:53:58.498041 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.497973    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1"} err="failed to get container status \"134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1\": rpc error: code = NotFound desc = could not find container \"134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1\": container with ID starting with 134aabce639e2dae1cdf605597452436ec699257d6a25e89723aa1f2417909d1 not found: ID does not exist"
Jul 09 18:53:58.513207 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.513165    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz" podStartSLOduration=27.623966491 podStartE2EDuration="37.513150357s" totalImagesPullingTime="9.889183866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:58.512295555 +0000 UTC m=+48.864730624" watchObservedRunningTime="2026-07-09 18:53:58.513150357 +0000 UTC m=+48.865585423"
Jul 09 18:53:58.582775 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.582720    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-869dcfcdcd-ncr7h" podStartSLOduration=27.688804236 podStartE2EDuration="37.582700089s" totalImagesPullingTime="9.893895853s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:58.58251923 +0000 UTC m=+48.934954294" watchObservedRunningTime="2026-07-09 18:53:58.582700089 +0000 UTC m=+48.935135156"
Jul 09 18:53:58.583300 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.583272    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-ltkwk" podStartSLOduration=2.6728446310000002 podStartE2EDuration="12.583259705s" totalImagesPullingTime="9.910415074s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:53:58.55246498 +0000 UTC m=+48.904900046" watchObservedRunningTime="2026-07-09 18:53:58.583259705 +0000 UTC m=+48.935694773"
Jul 09 18:53:58.636552 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.636449    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"]
Jul 09 18:53:58.650383 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.650354    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"
Jul 09 18:53:58.657557 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.655415    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 09 18:53:58.657557 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.655703    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 09 18:53:58.657557 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.655893    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-zjsm4\""
Jul 09 18:53:58.657557 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.657012    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"]
Jul 09 18:53:58.662341 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.662313    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:58.681291 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.681263    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq885\" (UniqueName: \"kubernetes.io/projected/849a8bde-788c-449a-bdc3-a7af39d97219-kube-api-access-vq885\") pod \"migrator-7f59b54f6c-xg7mj\" (UID: \"849a8bde-788c-449a-bdc3-a7af39d97219\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"
Jul 09 18:53:58.762730 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.762641    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:58.781799 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.781760    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq885\" (UniqueName: \"kubernetes.io/projected/849a8bde-788c-449a-bdc3-a7af39d97219-kube-api-access-vq885\") pod \"migrator-7f59b54f6c-xg7mj\" (UID: \"849a8bde-788c-449a-bdc3-a7af39d97219\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"
Jul 09 18:53:58.793956 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.793924    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq885\" (UniqueName: \"kubernetes.io/projected/849a8bde-788c-449a-bdc3-a7af39d97219-kube-api-access-vq885\") pod \"migrator-7f59b54f6c-xg7mj\" (UID: \"849a8bde-788c-449a-bdc3-a7af39d97219\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"
Jul 09 18:53:58.871135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.871077    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-sq65v"]
Jul 09 18:53:58.881144 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.881074    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:58.887010 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.886988    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:58.891459 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.891426    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 09 18:53:58.891725 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.891709    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 09 18:53:58.892059 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.891713    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k2d2w\""
Jul 09 18:53:58.894876 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.894806    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sq65v"]
Jul 09 18:53:58.964038 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.963993    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"
Jul 09 18:53:58.972346 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.972307    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:58.984596 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.984567    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:58.984717 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.984622    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:58.984717 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.984659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpt8k\" (UniqueName: \"kubernetes.io/projected/c0e38406-9aca-45ae-9792-e191317edbf3-kube-api-access-dpt8k\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:58.984717 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.984693    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0e38406-9aca-45ae-9792-e191317edbf3-data-volume\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:58.984847 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:58.984733    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0e38406-9aca-45ae-9792-e191317edbf3-crio-socket\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.087565 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.087514    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.087731 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.087605    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.087731 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.087637    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpt8k\" (UniqueName: \"kubernetes.io/projected/c0e38406-9aca-45ae-9792-e191317edbf3-kube-api-access-dpt8k\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.087731 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.087653    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:53:59.087731 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.087672    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0e38406-9aca-45ae-9792-e191317edbf3-data-volume\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.087731 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.087713    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:53:59.087731 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.087730    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:59.58770892 +0000 UTC m=+49.940143987 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:53:59.088031 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.087755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0e38406-9aca-45ae-9792-e191317edbf3-crio-socket\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.088031 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.087767    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:53:59.587754527 +0000 UTC m=+49.940189579 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : configmap "kube-rbac-proxy" not found
Jul 09 18:53:59.088031 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.087813    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c0e38406-9aca-45ae-9792-e191317edbf3-crio-socket\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.088031 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.088026    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c0e38406-9aca-45ae-9792-e191317edbf3-data-volume\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.098490 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.098426    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpt8k\" (UniqueName: \"kubernetes.io/projected/c0e38406-9aca-45ae-9792-e191317edbf3-kube-api-access-dpt8k\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.126478 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.126320    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj"]
Jul 09 18:53:59.128872 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:53:59.128843    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod849a8bde_788c_449a_bdc3_a7af39d97219.slice/crio-6d815e5e857fa58ba7d2093f9c6f025e194d9afdc291135eac63280f4c17e8bd WatchSource:0}: Error finding container 6d815e5e857fa58ba7d2093f9c6f025e194d9afdc291135eac63280f4c17e8bd: Status 404 returned error can't find the container with id 6d815e5e857fa58ba7d2093f9c6f025e194d9afdc291135eac63280f4c17e8bd
Jul 09 18:53:59.175013 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.174979    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:59.242719 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.242683    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:59.482361 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.482331    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ba3f622-358b-4925-a89d-3ac1d413b48e" containerID="6a88ab0bc00153c693c5f3f4f98fa5392097df3f5855b487c6532ec19ade39c6" exitCode=0
Jul 09 18:53:59.482770 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.482415    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerDied","Data":"6a88ab0bc00153c693c5f3f4f98fa5392097df3f5855b487c6532ec19ade39c6"}
Jul 09 18:53:59.483900 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.483884    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 18:53:59.484161 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.484145    2556 scope.go:117] "RemoveContainer" containerID="5d1cfaaa41fbc3d4ef0b3452655d3f46ccb7753a98f49227a7510ff652c4c06c"
Jul 09 18:53:59.484342 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.484321    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-9j4h9_openshift-console-operator(6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3)\"" pod="openshift-console-operator/console-operator-7866497494-9j4h9" podUID="6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3"
Jul 09 18:53:59.484933 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.484914    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj" event={"ID":"849a8bde-788c-449a-bdc3-a7af39d97219","Type":"ContainerStarted","Data":"6d815e5e857fa58ba7d2093f9c6f025e194d9afdc291135eac63280f4c17e8bd"}
Jul 09 18:53:59.592066 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.592032    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.592202 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.592112    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:53:59.592202 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.592139    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:53:59.592202 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.592180    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:00.592161276 +0000 UTC m=+50.944596318 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:53:59.592356 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.592205    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:53:59.592356 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:53:59.592287    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:00.592274743 +0000 UTC m=+50.944709787 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : configmap "kube-rbac-proxy" not found
Jul 09 18:53:59.646375 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.646334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:59.746464 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.746379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:53:59.983235 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:53:59.983199    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:00.087731 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.087656    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:00.492262 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.492228    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" event={"ID":"1ba3f622-358b-4925-a89d-3ac1d413b48e","Type":"ContainerStarted","Data":"f67822e48a347f839945401d4bab6744e1b16c04df733f267f97e639b2c49bf7"}
Jul 09 18:54:00.540609 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.540564    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-v4qmq" podStartSLOduration=9.06892712 podStartE2EDuration="50.540547922s" totalImagesPullingTime="41.471620802s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:00.52837964 +0000 UTC m=+50.880814709" watchObservedRunningTime="2026-07-09 18:54:00.540547922 +0000 UTC m=+50.892982986"
Jul 09 18:54:00.602049 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.602018    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:00.602173 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:00.602151    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:54:00.602230 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:00.602220    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.602200207 +0000 UTC m=+52.954635256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : configmap "kube-rbac-proxy" not found
Jul 09 18:54:00.602382 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.602358    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:00.602449 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:00.602435    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:54:00.602489 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:00.602479    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:02.602464167 +0000 UTC m=+52.954899219 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:54:00.769547 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.769438    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:00.843382 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.843356    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jkf4m_58622854-61de-4acb-a671-1cd2aa16fc97/dns-node-resolver/0.log"
Jul 09 18:54:00.851427 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:00.851375    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:01.497405 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.497369    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj" event={"ID":"849a8bde-788c-449a-bdc3-a7af39d97219","Type":"ContainerStarted","Data":"081caca5bc4cf937f4029fc29f0bd7a974291b32f63bb8e224d4baa307512ee9"}
Jul 09 18:54:01.497847 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.497431    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj" event={"ID":"849a8bde-788c-449a-bdc3-a7af39d97219","Type":"ContainerStarted","Data":"b41f48a8a1b5a02ffe9d5f480026fcbbc3dcd537f6ff1715cc817cca0d8659d9"}
Jul 09 18:54:01.548523 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.548469    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-xg7mj" podStartSLOduration=2.235212717 podStartE2EDuration="3.548453952s" totalImagesPullingTime="1.313241235s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:01.545280105 +0000 UTC m=+51.897715171" watchObservedRunningTime="2026-07-09 18:54:01.548453952 +0000 UTC m=+51.900889054"
Jul 09 18:54:01.559407 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.559369    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-9g7v6"]
Jul 09 18:54:01.562108 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.562088    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.566417 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.566393    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"kube-root-ca.crt\""
Jul 09 18:54:01.566592 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.566512    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"service-ca-dockercfg-fw789\""
Jul 09 18:54:01.566725 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.566711    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-service-ca\"/\"signing-key\""
Jul 09 18:54:01.566790 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.566770    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"openshift-service-ca.crt\""
Jul 09 18:54:01.566790 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.566714    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-service-ca\"/\"signing-cabundle\""
Jul 09 18:54:01.587625 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.587595    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-9g7v6"]
Jul 09 18:54:01.612562 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.612486    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtql2\" (UniqueName: \"kubernetes.io/projected/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-kube-api-access-vtql2\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.612753 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.612667    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-signing-key\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.612878 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.612851    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-signing-cabundle\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.657759 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.657732    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9qhwv_b67611f8-3dcd-4a7f-8948-3ea4b324e439/node-ca/0.log"
Jul 09 18:54:01.685707 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.685671    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:01.713513 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.713481    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtql2\" (UniqueName: \"kubernetes.io/projected/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-kube-api-access-vtql2\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.713692 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.713583    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-signing-key\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.713692 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.713655    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-signing-cabundle\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.714299 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.714281    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-cabundle\" (UniqueName: \"kubernetes.io/configmap/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-signing-cabundle\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.716023 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.715993    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"signing-key\" (UniqueName: \"kubernetes.io/secret/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-signing-key\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.726615 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.726581    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtql2\" (UniqueName: \"kubernetes.io/projected/4832db15-7c0f-4208-9efe-c8ab3d2aefe7-kube-api-access-vtql2\") pod \"service-ca-79f68b9f4f-9g7v6\" (UID: \"4832db15-7c0f-4208-9efe-c8ab3d2aefe7\") " pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:01.811711 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.811623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:01.870685 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:01.870638    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6"
Jul 09 18:54:02.011933 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.011900    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-service-ca/service-ca-79f68b9f4f-9g7v6"]
Jul 09 18:54:02.014656 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:02.014632    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4832db15_7c0f_4208_9efe_c8ab3d2aefe7.slice/crio-979ba75ea51441cb0028ce07670a67a6f726bba14b7e1c2d535bf0c14f472b6f WatchSource:0}: Error finding container 979ba75ea51441cb0028ce07670a67a6f726bba14b7e1c2d535bf0c14f472b6f: Status 404 returned error can't find the container with id 979ba75ea51441cb0028ce07670a67a6f726bba14b7e1c2d535bf0c14f472b6f
Jul 09 18:54:02.060224 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.060178    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:02.155053 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.155017    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:02.245256 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.245220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:02.420687 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.420651    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:54:02.420869 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.420701    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:54:02.420869 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.420774    2556 secret.go:189] Couldn't get secret openshift-monitoring/cluster-monitoring-operator-tls: secret "cluster-monitoring-operator-tls" not found
Jul 09 18:54:02.420869 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.420836    2556 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 09 18:54:02.420869 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.420854    2556 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7c4b4f9d7-n2cgw: secret "image-registry-tls" not found
Jul 09 18:54:02.421024 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.420856    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls podName:bc39ed10-3fcd-47e9-98af-2940484eb9ca nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.420835636 +0000 UTC m=+68.773270692 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "cluster-monitoring-operator-tls" (UniqueName: "kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls") pod "cluster-monitoring-operator-5f7f494dd4-rbkwz" (UID: "bc39ed10-3fcd-47e9-98af-2940484eb9ca") : secret "cluster-monitoring-operator-tls" not found
Jul 09 18:54:02.421024 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.420908    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls podName:0063f3df-5e0f-4576-8348-81aa821a2857 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.420891203 +0000 UTC m=+68.773326250 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls") pod "image-registry-7c4b4f9d7-n2cgw" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857") : secret "image-registry-tls" not found
Jul 09 18:54:02.462357 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.462328    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ltkwk_b49c8ad2-d3d4-44bd-91d1-d454284d0c64/serve-healthcheck-canary/0.log"
Jul 09 18:54:02.502071 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.502033    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6" event={"ID":"4832db15-7c0f-4208-9efe-c8ab3d2aefe7","Type":"ContainerStarted","Data":"2d7c3cfabf45f26acc38dbe4e7b15c4587f71a3b9ebfb2639588270f1f2b093e"}
Jul 09 18:54:02.502437 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.502096    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6" event={"ID":"4832db15-7c0f-4208-9efe-c8ab3d2aefe7","Type":"ContainerStarted","Data":"979ba75ea51441cb0028ce07670a67a6f726bba14b7e1c2d535bf0c14f472b6f"}
Jul 09 18:54:02.521494 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.521465    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:54:02.521628 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.521519    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:02.521628 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.521624    2556 secret.go:189] Couldn't get secret openshift-ingress/router-metrics-certs-default: secret "router-metrics-certs-default" not found
Jul 09 18:54:02.521695 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.521629    2556 secret.go:189] Couldn't get secret openshift-cluster-samples-operator/samples-operator-tls: secret "samples-operator-tls" not found
Jul 09 18:54:02.521695 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.521684    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.521664054 +0000 UTC m=+68.874099104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : secret "router-metrics-certs-default" not found
Jul 09 18:54:02.521767 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.521708    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls podName:92ee9458-91f1-4c04-8144-29691b182b9d nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.521700887 +0000 UTC m=+68.874135930 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "samples-operator-tls" (UniqueName: "kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls") pod "cluster-samples-operator-65db858894-zjtpg" (UID: "92ee9458-91f1-4c04-8144-29691b182b9d") : secret "samples-operator-tls" not found
Jul 09 18:54:02.521812 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.521797    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:02.521985 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.521970    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle podName:7d5933ef-2952-4201-891b-78f29352a829 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.521953423 +0000 UTC m=+68.874388477 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "service-ca-bundle" (UniqueName: "kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle") pod "router-default-66ddf58c96-f6w75" (UID: "7d5933ef-2952-4201-891b-78f29352a829") : configmap references non-existent config key: service-ca.crt
Jul 09 18:54:02.537113 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.537064    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-service-ca/service-ca-79f68b9f4f-9g7v6" podStartSLOduration=1.537048322 podStartE2EDuration="1.537048322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:02.535761519 +0000 UTC m=+52.888196583" watchObservedRunningTime="2026-07-09 18:54:02.537048322 +0000 UTC m=+52.889483388"
Jul 09 18:54:02.538757 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.538731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:02.622744 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.622704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:54:02.622938 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.622755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:54:02.622938 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.622798    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:02.622938 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.622836    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:02.622938 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.622885    2556 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 09 18:54:02.622938 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.622905    2556 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Jul 09 18:54:02.623180 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.622956    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls podName:2000c44b-544b-4072-8d0a-604d4b1e1d47 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.622934074 +0000 UTC m=+68.975369120 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls") pod "dns-default-5knnr" (UID: "2000c44b-544b-4072-8d0a-604d4b1e1d47") : secret "dns-default-metrics-tls" not found
Jul 09 18:54:02.623180 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.622975    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert podName:0208420e-d48a-418d-b036-9a1a611ad6bf nodeName:}" failed. No retries permitted until 2026-07-09 18:54:18.622962896 +0000 UTC m=+68.975397941 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert") pod "networking-console-plugin-6cf856758c-t2zpb" (UID: "0208420e-d48a-418d-b036-9a1a611ad6bf") : secret "networking-console-plugin-cert" not found
Jul 09 18:54:02.623180 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.623035    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:54:02.623180 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.623068    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:06.623058482 +0000 UTC m=+56.975493527 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:54:02.623180 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.623096    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:54:02.623180 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:02.623150    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:06.623134696 +0000 UTC m=+56.975569744 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : configmap "kube-rbac-proxy" not found
Jul 09 18:54:02.657594 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.657559    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xg7mj_849a8bde-788c-449a-bdc3-a7af39d97219/migrator/0.log"
Jul 09 18:54:02.850516 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:02.850431    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xg7mj_849a8bde-788c-449a-bdc3-a7af39d97219/graceful-termination/0.log"
Jul 09 18:54:03.021296 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:03.021259    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:03.053251 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:03.053181    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-rgrvz_ff7c7467-a4b1-44aa-8dd8-50ed702d7f87/kube-storage-version-migrator-operator/0.log"
Jul 09 18:54:03.321873 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:03.321838    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:03.440174 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:03.440138    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:03.560844 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:03.560813    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:03.679806 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:03.679770    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:03.801892 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:03.801860    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:04.792714 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:04.792668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:05.007890 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:05.007851    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:05.393261 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:05.393229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:05.587716 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:05.587683    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:06.238139 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.238101    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:06.575764 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.575679    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:06.663211 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.663176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:06.663369 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.663219    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:06.663369 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:06.663318    2556 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Jul 09 18:54:06.663369 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:06.663324    2556 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 09 18:54:06.663469 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:06.663385    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:14.663366126 +0000 UTC m=+65.015801173 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : secret "insights-runtime-extractor-tls" not found
Jul 09 18:54:06.663469 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:06.663402    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm podName:c0e38406-9aca-45ae-9792-e191317edbf3 nodeName:}" failed. No retries permitted until 2026-07-09 18:54:14.663395097 +0000 UTC m=+65.015830140 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm") pod "insights-runtime-extractor-sq65v" (UID: "c0e38406-9aca-45ae-9792-e191317edbf3") : configmap "kube-rbac-proxy" not found
Jul 09 18:54:06.683963 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.683928    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:06.925445 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.925416    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:54:06.925691 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.925468    2556 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:54:06.925842 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:06.925827    2556 scope.go:117] "RemoveContainer" containerID="5d1cfaaa41fbc3d4ef0b3452655d3f46ccb7753a98f49227a7510ff652c4c06c"
Jul 09 18:54:06.925991 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:54:06.925973    2556 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"console-operator\" with CrashLoopBackOff: \"back-off 10s restarting failed container=console-operator pod=console-operator-7866497494-9j4h9_openshift-console-operator(6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3)\"" pod="openshift-console-operator/console-operator-7866497494-9j4h9" podUID="6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3"
Jul 09 18:54:07.002941 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:07.002899    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:07.109663 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:07.109630    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:07.251484 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:07.247747    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:07.797264 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:07.797227    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:07.890845 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:07.890812    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:08.376407 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:08.376366    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:08.588925 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:08.588890    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:08.793067 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:08.793037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:08.990010 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:08.989977    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:09.285186 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:09.285149    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:09.970186 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:09.970150    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:10.371949 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:10.371848    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:10.580336 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:10.580293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:10.775797 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:10.775766    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:11.174354 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:11.174320    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:11.781270 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:11.781229    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:12.170495 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:12.170463    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:12.603630 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:12.603540    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:12.769076 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:12.769046    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:13.168813 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:13.168779    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:13.401772 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:13.401750    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-kbcj5"
Jul 09 18:54:13.496451 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:13.496373    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:13.597624 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:13.597585    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:13.697140 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:13.697100    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:13.802477 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:13.802379    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:14.739660 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:14.739607    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:14.740137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:14.739855    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:14.740137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:14.740126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c0e38406-9aca-45ae-9792-e191317edbf3-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:14.742151 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:14.742128    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c0e38406-9aca-45ae-9792-e191317edbf3-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-sq65v\" (UID: \"c0e38406-9aca-45ae-9792-e191317edbf3\") " pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:14.800777 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:14.800748    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-k2d2w\""
Jul 09 18:54:14.808722 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:14.808704    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-sq65v"
Jul 09 18:54:14.934031 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:14.934009    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-sq65v"]
Jul 09 18:54:14.936207 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:14.936184    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0e38406_9aca_45ae_9792_e191317edbf3.slice/crio-68f61e2872de87bed38d29a4e1e298970c4e24c81fe606c948419315a644f0e9 WatchSource:0}: Error finding container 68f61e2872de87bed38d29a4e1e298970c4e24c81fe606c948419315a644f0e9: Status 404 returned error can't find the container with id 68f61e2872de87bed38d29a4e1e298970c4e24c81fe606c948419315a644f0e9
Jul 09 18:54:15.553414 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.553326    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq65v" event={"ID":"c0e38406-9aca-45ae-9792-e191317edbf3","Type":"ContainerStarted","Data":"68f61e2872de87bed38d29a4e1e298970c4e24c81fe606c948419315a644f0e9"}
Jul 09 18:54:15.750136 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.750099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:54:15.753472 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.753446    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 09 18:54:15.765767 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.765739    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/f721565e-88df-4cad-b2e4-49193dcfbddd-original-pull-secret\") pod \"global-pull-secret-syncer-v9667\" (UID: \"f721565e-88df-4cad-b2e4-49193dcfbddd\") " pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:54:15.799815 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.799754    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:15.851103 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.851005    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:54:15.854763 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.854737    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 09 18:54:15.865194 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.865167    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8a09070-edcc-43d2-8b08-a3fd1950f284-metrics-certs\") pod \"network-metrics-daemon-66mk7\" (UID: \"a8a09070-edcc-43d2-8b08-a3fd1950f284\") " pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:54:15.910970 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.910941    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-v9667"
Jul 09 18:54:15.951929 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.951896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:54:15.954916 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:15.954850    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzxl9\" (UniqueName: \"kubernetes.io/projected/e99ad357-fdc9-4571-9528-e5743907647a-kube-api-access-hzxl9\") pod \"network-check-target-rs466\" (UID: \"e99ad357-fdc9-4571-9528-e5743907647a\") " pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:54:16.005831 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.005801    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qkpmb\""
Jul 09 18:54:16.012398 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.012359    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-66mk7"
Jul 09 18:54:16.016042 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.015980    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-47bt6\""
Jul 09 18:54:16.024484 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.024456    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:54:16.061463 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.061378    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-v9667"]
Jul 09 18:54:16.066414 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:16.066376    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf721565e_88df_4cad_b2e4_49193dcfbddd.slice/crio-83756aff68322cd5ffa3140baece576551ef38d8750dbf3ebec8a908193e5da3 WatchSource:0}: Error finding container 83756aff68322cd5ffa3140baece576551ef38d8750dbf3ebec8a908193e5da3: Status 404 returned error can't find the container with id 83756aff68322cd5ffa3140baece576551ef38d8750dbf3ebec8a908193e5da3
Jul 09 18:54:16.109808 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.109714    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:16.167671 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.167645    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-66mk7"]
Jul 09 18:54:16.170406 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:16.170375    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8a09070_edcc_43d2_8b08_a3fd1950f284.slice/crio-968ba64cd4610b214bff98e14f4e031744b87af61aa5c999e71b5acb1535f7af WatchSource:0}: Error finding container 968ba64cd4610b214bff98e14f4e031744b87af61aa5c999e71b5acb1535f7af: Status 404 returned error can't find the container with id 968ba64cd4610b214bff98e14f4e031744b87af61aa5c999e71b5acb1535f7af
Jul 09 18:54:16.193927 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.193900    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-rs466"]
Jul 09 18:54:16.196758 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:16.196733    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode99ad357_fdc9_4571_9528_e5743907647a.slice/crio-eb2dfe123f4c76d928ab531cf5e356c1122de1a59830fc6c4f5cbd8252cab9a2 WatchSource:0}: Error finding container eb2dfe123f4c76d928ab531cf5e356c1122de1a59830fc6c4f5cbd8252cab9a2: Status 404 returned error can't find the container with id eb2dfe123f4c76d928ab531cf5e356c1122de1a59830fc6c4f5cbd8252cab9a2
Jul 09 18:54:16.232674 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.232612    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:16.559745 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.559712    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rs466" event={"ID":"e99ad357-fdc9-4571-9528-e5743907647a","Type":"ContainerStarted","Data":"eca793380004b3a53534826e18653076257d0970a38660a72f522a4eb65e1a83"}
Jul 09 18:54:16.559930 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.559766    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-rs466" event={"ID":"e99ad357-fdc9-4571-9528-e5743907647a","Type":"ContainerStarted","Data":"eb2dfe123f4c76d928ab531cf5e356c1122de1a59830fc6c4f5cbd8252cab9a2"}
Jul 09 18:54:16.559930 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.559828    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:54:16.561027 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.560993    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66mk7" event={"ID":"a8a09070-edcc-43d2-8b08-a3fd1950f284","Type":"ContainerStarted","Data":"968ba64cd4610b214bff98e14f4e031744b87af61aa5c999e71b5acb1535f7af"}
Jul 09 18:54:16.562116 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.562092    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v9667" event={"ID":"f721565e-88df-4cad-b2e4-49193dcfbddd","Type":"ContainerStarted","Data":"83756aff68322cd5ffa3140baece576551ef38d8750dbf3ebec8a908193e5da3"}
Jul 09 18:54:16.580319 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:16.580206    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-rs466" podStartSLOduration=66.580186008 podStartE2EDuration="1m6.580186008s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:16.580011434 +0000 UTC m=+66.932446502" watchObservedRunningTime="2026-07-09 18:54:16.580186008 +0000 UTC m=+66.932621074"
Jul 09 18:54:18.475539 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.475477    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:54:18.476005 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.475549    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:54:18.478463 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.478434    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"image-registry-7c4b4f9d7-n2cgw\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") " pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:54:18.478973 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.478950    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-monitoring-operator-tls\" (UniqueName: \"kubernetes.io/secret/bc39ed10-3fcd-47e9-98af-2940484eb9ca-cluster-monitoring-operator-tls\") pod \"cluster-monitoring-operator-5f7f494dd4-rbkwz\" (UID: \"bc39ed10-3fcd-47e9-98af-2940484eb9ca\") " pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:54:18.577021 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.576993    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:54:18.577021 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.577028    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:18.577244 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.577102    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:18.577772 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.577743    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/7d5933ef-2952-4201-891b-78f29352a829-service-ca-bundle\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:18.579707 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.579686    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"samples-operator-tls\" (UniqueName: \"kubernetes.io/secret/92ee9458-91f1-4c04-8144-29691b182b9d-samples-operator-tls\") pod \"cluster-samples-operator-65db858894-zjtpg\" (UID: \"92ee9458-91f1-4c04-8144-29691b182b9d\") " pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:54:18.579864 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.579843    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7d5933ef-2952-4201-891b-78f29352a829-metrics-certs\") pod \"router-default-66ddf58c96-f6w75\" (UID: \"7d5933ef-2952-4201-891b-78f29352a829\") " pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:18.670746 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.670715    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-4w25g\""
Jul 09 18:54:18.677544 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.677479    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:54:18.677792 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.677766    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:54:18.677874 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.677817    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:54:18.680568 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.680372    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"cluster-monitoring-operator-dockercfg-strwc\""
Jul 09 18:54:18.681180 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.680854    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/2000c44b-544b-4072-8d0a-604d4b1e1d47-metrics-tls\") pod \"dns-default-5knnr\" (UID: \"2000c44b-544b-4072-8d0a-604d4b1e1d47\") " pod="openshift-dns/dns-default-5knnr"
Jul 09 18:54:18.681391 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.681349    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/0208420e-d48a-418d-b036-9a1a611ad6bf-networking-console-plugin-cert\") pod \"networking-console-plugin-6cf856758c-t2zpb\" (UID: \"0208420e-d48a-418d-b036-9a1a611ad6bf\") " pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:54:18.687476 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.687295    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"
Jul 09 18:54:18.696126 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.696105    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-samples-operator\"/\"cluster-samples-operator-dockercfg-cmpq6\""
Jul 09 18:54:18.700783 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.700758    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:18.704596 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.704539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"
Jul 09 18:54:18.704983 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.704963    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"router-dockercfg-zrs78\""
Jul 09 18:54:18.713614 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.713483    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:18.797648 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.797568    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xvss5\""
Jul 09 18:54:18.805036 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.804965    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-5knnr"
Jul 09 18:54:18.825280 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.825250    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-szg6x\""
Jul 09 18:54:18.832599 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:18.832576    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"
Jul 09 18:54:20.838683 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:20.838656    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/router-default-66ddf58c96-f6w75"]
Jul 09 18:54:20.876309 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:20.875950    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"]
Jul 09 18:54:20.888365 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:20.888325    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0063f3df_5e0f_4576_8348_81aa821a2857.slice/crio-a07a350fd830c60a791e478c675877f6ceb3d0c71f0c3e494f0ffc44136a4fe0 WatchSource:0}: Error finding container a07a350fd830c60a791e478c675877f6ceb3d0c71f0c3e494f0ffc44136a4fe0: Status 404 returned error can't find the container with id a07a350fd830c60a791e478c675877f6ceb3d0c71f0c3e494f0ffc44136a4fe0
Jul 09 18:54:21.079069 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.079045    2556 scope.go:117] "RemoveContainer" containerID="5d1cfaaa41fbc3d4ef0b3452655d3f46ccb7753a98f49227a7510ff652c4c06c"
Jul 09 18:54:21.099894 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.099873    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-5knnr"]
Jul 09 18:54:21.103371 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:21.103350    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2000c44b_544b_4072_8d0a_604d4b1e1d47.slice/crio-48fb30ad22951b37521b2b80a38e3197aaeb6cdd78fac4071e75229d594ece46 WatchSource:0}: Error finding container 48fb30ad22951b37521b2b80a38e3197aaeb6cdd78fac4071e75229d594ece46: Status 404 returned error can't find the container with id 48fb30ad22951b37521b2b80a38e3197aaeb6cdd78fac4071e75229d594ece46
Jul 09 18:54:21.103745 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.103724    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz"]
Jul 09 18:54:21.105417 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.105160    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg"]
Jul 09 18:54:21.106319 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.106298    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6cf856758c-t2zpb"]
Jul 09 18:54:21.109294 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:21.109268    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc39ed10_3fcd_47e9_98af_2940484eb9ca.slice/crio-afaa7eefbfb46a904d233459c0e36e031afd403eeee9f8634e45939b16031083 WatchSource:0}: Error finding container afaa7eefbfb46a904d233459c0e36e031afd403eeee9f8634e45939b16031083: Status 404 returned error can't find the container with id afaa7eefbfb46a904d233459c0e36e031afd403eeee9f8634e45939b16031083
Jul 09 18:54:21.109991 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:21.109964    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0208420e_d48a_418d_b036_9a1a611ad6bf.slice/crio-f9f04c0acc0a392747c478c95201ef7f3bc9912ce71b3cc22cd135b81145b04f WatchSource:0}: Error finding container f9f04c0acc0a392747c478c95201ef7f3bc9912ce71b3cc22cd135b81145b04f: Status 404 returned error can't find the container with id f9f04c0acc0a392747c478c95201ef7f3bc9912ce71b3cc22cd135b81145b04f
Jul 09 18:54:21.585164 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.585138    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 18:54:21.585283 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.585262    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console-operator/console-operator-7866497494-9j4h9" event={"ID":"6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3","Type":"ContainerStarted","Data":"fd3cf7461322619d8349b850b81ed2b7d49d4b20ebc6623f683d80b0dd84b468"}
Jul 09 18:54:21.585846 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.585827    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:54:21.587332 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.587249    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5knnr" event={"ID":"2000c44b-544b-4072-8d0a-604d4b1e1d47","Type":"ContainerStarted","Data":"48fb30ad22951b37521b2b80a38e3197aaeb6cdd78fac4071e75229d594ece46"}
Jul 09 18:54:21.588510 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.588485    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz" event={"ID":"bc39ed10-3fcd-47e9-98af-2940484eb9ca","Type":"ContainerStarted","Data":"afaa7eefbfb46a904d233459c0e36e031afd403eeee9f8634e45939b16031083"}
Jul 09 18:54:21.589929 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.589907    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg" event={"ID":"92ee9458-91f1-4c04-8144-29691b182b9d","Type":"ContainerStarted","Data":"104be52bd8c92ae9e2909ab5191cf6725e0d01f53181ab7ef2493a2bd57709f5"}
Jul 09 18:54:21.592011 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.591989    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq65v" event={"ID":"c0e38406-9aca-45ae-9792-e191317edbf3","Type":"ContainerStarted","Data":"5dbaaa992c73173ff14e6b1e420111a1fdc84cb81571157993abcee6c02b7748"}
Jul 09 18:54:21.593627 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.593602    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw" event={"ID":"0063f3df-5e0f-4576-8348-81aa821a2857","Type":"ContainerStarted","Data":"aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823"}
Jul 09 18:54:21.593710 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.593640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw" event={"ID":"0063f3df-5e0f-4576-8348-81aa821a2857","Type":"ContainerStarted","Data":"a07a350fd830c60a791e478c675877f6ceb3d0c71f0c3e494f0ffc44136a4fe0"}
Jul 09 18:54:21.593710 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.593665    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:54:21.595607 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.595584    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66ddf58c96-f6w75" event={"ID":"7d5933ef-2952-4201-891b-78f29352a829","Type":"ContainerStarted","Data":"dd4a48e7d2e54a5a61b55b0f52f9d091a7720aa806af1afa16b591dd851d398d"}
Jul 09 18:54:21.595698 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.595627    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/router-default-66ddf58c96-f6w75" event={"ID":"7d5933ef-2952-4201-891b-78f29352a829","Type":"ContainerStarted","Data":"57e4cf08a63c4c1d1e28ce4d3a0f45ac6992aaf395e3989c09d8511b93f6d99f"}
Jul 09 18:54:21.596791 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.596771    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb" event={"ID":"0208420e-d48a-418d-b036-9a1a611ad6bf","Type":"ContainerStarted","Data":"f9f04c0acc0a392747c478c95201ef7f3bc9912ce71b3cc22cd135b81145b04f"}
Jul 09 18:54:21.598801 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.598778    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66mk7" event={"ID":"a8a09070-edcc-43d2-8b08-a3fd1950f284","Type":"ContainerStarted","Data":"bbe8bf002fb943f994e77ddb038b841ab28bc10790fa8391ad26b1f4359aa02d"}
Jul 09 18:54:21.598890 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.598823    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-66mk7" event={"ID":"a8a09070-edcc-43d2-8b08-a3fd1950f284","Type":"ContainerStarted","Data":"8bf167d1d26f0104c5b2afe2dc41b205331341e8a85f0dca6220ab47ff956491"}
Jul 09 18:54:21.601166 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.601145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-v9667" event={"ID":"f721565e-88df-4cad-b2e4-49193dcfbddd","Type":"ContainerStarted","Data":"2127a05947657b1423c9f9ea870735262292fe1517f13ead7cb2c6743d8fcff6"}
Jul 09 18:54:21.605000 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.604960    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console-operator/console-operator-7866497494-9j4h9" podStartSLOduration=50.634262098 podStartE2EDuration="1m0.604947411s" totalImagesPullingTime="9.970685313s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:21.604281217 +0000 UTC m=+71.956716283" watchObservedRunningTime="2026-07-09 18:54:21.604947411 +0000 UTC m=+71.957382467"
Jul 09 18:54:21.621280 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.621086    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-66mk7" podStartSLOduration=67.128884036 podStartE2EDuration="1m11.621069432s" totalImagesPullingTime="4.492185396s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:21.620146154 +0000 UTC m=+71.972581305" watchObservedRunningTime="2026-07-09 18:54:21.621069432 +0000 UTC m=+71.973504498"
Jul 09 18:54:21.639799 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.639756    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw" podStartSLOduration=71.639743157 podStartE2EDuration="1m11.639743157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:21.638485428 +0000 UTC m=+71.990920494" watchObservedRunningTime="2026-07-09 18:54:21.639743157 +0000 UTC m=+71.992178221"
Jul 09 18:54:21.658813 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.658750    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/router-default-66ddf58c96-f6w75" podStartSLOduration=60.658736918 podStartE2EDuration="1m0.658736918s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:21.657568662 +0000 UTC m=+72.010003730" watchObservedRunningTime="2026-07-09 18:54:21.658736918 +0000 UTC m=+72.011171982"
Jul 09 18:54:21.672826 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.672777    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-v9667" podStartSLOduration=67.168883256 podStartE2EDuration="1m11.672763786s" totalImagesPullingTime="4.50388053s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:21.672287042 +0000 UTC m=+72.024722108" watchObservedRunningTime="2026-07-09 18:54:21.672763786 +0000 UTC m=+72.025198851"
Jul 09 18:54:21.714074 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.713973    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:21.717430 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:21.717238    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:22.160468 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.160236    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console-operator/console-operator-7866497494-9j4h9"
Jul 09 18:54:22.363955 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.363919    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:22.462443 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.462384    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:22.549949 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.549895    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:22.620397 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.620358    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq65v" event={"ID":"c0e38406-9aca-45ae-9792-e191317edbf3","Type":"ContainerStarted","Data":"dfa53a2fb2ed18a4a0ccd13243ba5969e29e14be6a7e4377aad3c65d35d2bc52"}
Jul 09 18:54:22.621964 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.621916    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:22.623730 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.623703    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/router-default-66ddf58c96-f6w75"
Jul 09 18:54:22.637919 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:22.637886    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:23.012518 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:23.012482    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:23.114555 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:23.114476    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:23.420319 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:23.420288    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:23.521608 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:23.521573    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:23.881302 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:23.881201    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:24.051161 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:24.051129    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:24.245911 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:24.245882    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:24.593027 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:24.592942    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:25.495487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:25.495436    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:25.582872 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:25.582836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:27.264826 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.264761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:27.370508 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.370400    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:27.472050 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.472011    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:27.640154 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.640068    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5knnr" event={"ID":"2000c44b-544b-4072-8d0a-604d4b1e1d47","Type":"ContainerStarted","Data":"9fb3c0135e6db43c9b9764373b38465d78440edcce09edb9b3385b256f995d54"}
Jul 09 18:54:27.640325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.640156    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-5knnr" event={"ID":"2000c44b-544b-4072-8d0a-604d4b1e1d47","Type":"ContainerStarted","Data":"627a5335aa6f1c3e328f2114c0210dd872217d5ec17f1df7b96b8b7c6f2e5386"}
Jul 09 18:54:27.640325 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.640190    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-5knnr"
Jul 09 18:54:27.641708 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.641674    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz" event={"ID":"bc39ed10-3fcd-47e9-98af-2940484eb9ca","Type":"ContainerStarted","Data":"a99cf818989d9e23646b921f13081bb48dd2c625e9e18fcd6dcc21b1818fc167"}
Jul 09 18:54:27.643609 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.643588    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg" event={"ID":"92ee9458-91f1-4c04-8144-29691b182b9d","Type":"ContainerStarted","Data":"b75d5cae6a5c2fa96a2eb1efdef9607808331792e7d286501a2595854f6684d2"}
Jul 09 18:54:27.643758 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.643742    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg" event={"ID":"92ee9458-91f1-4c04-8144-29691b182b9d","Type":"ContainerStarted","Data":"6de6af1967be973b7e2b29104a674aeefe5240f565418deeb5f75d9ad87e5d10"}
Jul 09 18:54:27.645824 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.645797    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-sq65v" event={"ID":"c0e38406-9aca-45ae-9792-e191317edbf3","Type":"ContainerStarted","Data":"b2b353ad08cdf81cafd87b8caff4ae4e1e65509dea9742c6b515802da12cfa97"}
Jul 09 18:54:27.647226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.647208    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb" event={"ID":"0208420e-d48a-418d-b036-9a1a611ad6bf","Type":"ContainerStarted","Data":"b0d7653887925e40291891431e7124b35e47ef0999a17e61900429e3e7e4ccbd"}
Jul 09 18:54:27.657314 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.657265    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-5knnr" podStartSLOduration=35.872166462 podStartE2EDuration="41.657250116s" totalImagesPullingTime="5.785083654s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:27.656546356 +0000 UTC m=+78.008981425" watchObservedRunningTime="2026-07-09 18:54:27.657250116 +0000 UTC m=+78.009685263"
Jul 09 18:54:27.674749 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.674679    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/cluster-monitoring-operator-5f7f494dd4-rbkwz" podStartSLOduration=60.89593698 podStartE2EDuration="1m6.674662492s" totalImagesPullingTime="5.778725512s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:27.673469135 +0000 UTC m=+78.025904196" watchObservedRunningTime="2026-07-09 18:54:27.674662492 +0000 UTC m=+78.027097567"
Jul 09 18:54:27.692782 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.692729    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6cf856758c-t2zpb" podStartSLOduration=55.914451643 podStartE2EDuration="1m1.692711207s" totalImagesPullingTime="5.778259564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:27.692461116 +0000 UTC m=+78.044896182" watchObservedRunningTime="2026-07-09 18:54:27.692711207 +0000 UTC m=+78.045146272"
Jul 09 18:54:27.712811 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.712760    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-samples-operator/cluster-samples-operator-65db858894-zjtpg" podStartSLOduration=60.971885312 podStartE2EDuration="1m6.712747307s" totalImagesPullingTime="5.740861995s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:53:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:27.711514608 +0000 UTC m=+78.063949676" watchObservedRunningTime="2026-07-09 18:54:27.712747307 +0000 UTC m=+78.065182371"
Jul 09 18:54:27.731674 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.731626    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-sq65v" podStartSLOduration=18.062970249 podStartE2EDuration="29.731610939s" totalImagesPullingTime="11.66864069s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 18:53:58 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:27.730985666 +0000 UTC m=+78.083420730" watchObservedRunningTime="2026-07-09 18:54:27.731610939 +0000 UTC m=+78.084046004"
Jul 09 18:54:27.756212 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:27.756176    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:28.497199 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:28.497165    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:28.557202 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:28.557166    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:33.669621 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.669479    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"]
Jul 09 18:54:33.678114 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.678082    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"]
Jul 09 18:54:33.678276 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.678218    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.680102 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.680081    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"]
Jul 09 18:54:33.680232 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.680197    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:33.681636 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.681617    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 09 18:54:33.682246 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.682230    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 09 18:54:33.682353 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.682288    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.682515 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.682498    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 09 18:54:33.682841 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.682823    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 09 18:54:33.682923 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.682827    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 09 18:54:33.684556 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.684516    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-ks42l\""
Jul 09 18:54:33.686673 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.686622    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 09 18:54:33.686673 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.686661    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 09 18:54:33.686841 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.686680    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 09 18:54:33.687068 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.687042    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 09 18:54:33.695139 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.695117    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"]
Jul 09 18:54:33.697115 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.697093    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"]
Jul 09 18:54:33.703297 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.703264    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"]
Jul 09 18:54:33.709025 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.708999    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93f41dd7-66bb-4673-9917-beddc88c6a7d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6\" (UID: \"93f41dd7-66bb-4673-9917-beddc88c6a7d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:33.709141 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dsn9t\" (UniqueName: \"kubernetes.io/projected/0d81ff97-5313-4df1-ac0a-6050815722e3-kube-api-access-dsn9t\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.709141 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709094    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ctbbd\" (UniqueName: \"kubernetes.io/projected/99777806-5fca-4857-96cc-db7dcb93da69-kube-api-access-ctbbd\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.709141 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709123    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99777806-5fca-4857-96cc-db7dcb93da69-tmp\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.709299 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709157    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zznmp\" (UniqueName: \"kubernetes.io/projected/93f41dd7-66bb-4673-9917-beddc88c6a7d-kube-api-access-zznmp\") pod \"managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6\" (UID: \"93f41dd7-66bb-4673-9917-beddc88c6a7d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:33.709299 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709239    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.709299 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709271    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-hub\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.709431 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709314    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-ca\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.709431 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709337    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/99777806-5fca-4857-96cc-db7dcb93da69-klusterlet-config\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.709431 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709364    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d81ff97-5313-4df1-ac0a-6050815722e3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.709543 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.709440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.770006 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.769976    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"]
Jul 09 18:54:33.772186 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.772169    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-68c9cbb497-npb4s"]
Jul 09 18:54:33.772302 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.772282    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"
Jul 09 18:54:33.774085 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.774058    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-npb4s"
Jul 09 18:54:33.775398 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.775381    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 09 18:54:33.775691 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.775677    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-24tdh\""
Jul 09 18:54:33.778558 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.778517    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Jul 09 18:54:33.778615 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.778559    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Jul 09 18:54:33.778615 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.778521    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-vphjx\""
Jul 09 18:54:33.785856 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.785837    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-579df8cdd4-7r7z5"]
Jul 09 18:54:33.788202 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.788183    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.789090 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.789064    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-npb4s"]
Jul 09 18:54:33.790255 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.790236    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"]
Jul 09 18:54:33.790680 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.790662    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cplfs\""
Jul 09 18:54:33.791013 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.790997    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 18:54:33.791109 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.790999    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 18:54:33.794994 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.794960    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 18:54:33.795157 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.794962    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 18:54:33.796394 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.796368    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 18:54:33.796590 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.796573    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:33.809999 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.809975    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-ca\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.810083 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810000    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/99777806-5fca-4857-96cc-db7dcb93da69-klusterlet-config\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.810083 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810021    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pj2gd\" (UniqueName: \"kubernetes.io/projected/572f9a0d-e008-40a3-8f92-6f989eebdb8f-kube-api-access-pj2gd\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.810083 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810041    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d81ff97-5313-4df1-ac0a-6050815722e3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.810230 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810217    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/407c46f4-31ca-4f90-a108-a2deab7ad827-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zxx77\" (UID: \"407c46f4-31ca-4f90-a108-a2deab7ad827\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"
Jul 09 18:54:33.810283 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810250    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.810346 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810282    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-oauth-config\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.810400 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810364    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93f41dd7-66bb-4673-9917-beddc88c6a7d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6\" (UID: \"93f41dd7-66bb-4673-9917-beddc88c6a7d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:33.810452 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810396    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-oauth-serving-cert\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.810452 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810427    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dsn9t\" (UniqueName: \"kubernetes.io/projected/0d81ff97-5313-4df1-ac0a-6050815722e3-kube-api-access-dsn9t\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.810582 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810451    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ctbbd\" (UniqueName: \"kubernetes.io/projected/99777806-5fca-4857-96cc-db7dcb93da69-kube-api-access-ctbbd\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.810582 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810484    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99777806-5fca-4857-96cc-db7dcb93da69-tmp\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.810582 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810510    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-config\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.810582 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810557    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zznmp\" (UniqueName: \"kubernetes.io/projected/93f41dd7-66bb-4673-9917-beddc88c6a7d-kube-api-access-zznmp\") pod \"managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6\" (UID: \"93f41dd7-66bb-4673-9917-beddc88c6a7d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:33.810809 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810599    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.810809 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810635    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-hub\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.810809 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-service-ca\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.810809 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810723    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srpqp\" (UniqueName: \"kubernetes.io/projected/ada802fe-ca8b-4f1d-a8aa-920a67908065-kube-api-access-srpqp\") pod \"downloads-68c9cbb497-npb4s\" (UID: \"ada802fe-ca8b-4f1d-a8aa-920a67908065\") " pod="openshift-console/downloads-68c9cbb497-npb4s"
Jul 09 18:54:33.810809 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810764    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-serving-cert\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.811031 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.810918    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/0d81ff97-5313-4df1-ac0a-6050815722e3-additional-service-ca\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.811300 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.811276    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/99777806-5fca-4857-96cc-db7dcb93da69-tmp\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.812813 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.812782    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-ca\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.815230 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.813838    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.815230 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.813909    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-hub\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.815230 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.813929    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/0d81ff97-5313-4df1-ac0a-6050815722e3-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.815230 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.813998    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/93f41dd7-66bb-4673-9917-beddc88c6a7d-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6\" (UID: \"93f41dd7-66bb-4673-9917-beddc88c6a7d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:33.820030 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.815916    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/99777806-5fca-4857-96cc-db7dcb93da69-klusterlet-config\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.825140 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.825122    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579df8cdd4-7r7z5"]
Jul 09 18:54:33.841725 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.841701    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dsn9t\" (UniqueName: \"kubernetes.io/projected/0d81ff97-5313-4df1-ac0a-6050815722e3-kube-api-access-dsn9t\") pod \"cluster-proxy-proxy-agent-5fcff995b-6bvr5\" (UID: \"0d81ff97-5313-4df1-ac0a-6050815722e3\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:33.842834 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.842818    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zznmp\" (UniqueName: \"kubernetes.io/projected/93f41dd7-66bb-4673-9917-beddc88c6a7d-kube-api-access-zznmp\") pod \"managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6\" (UID: \"93f41dd7-66bb-4673-9917-beddc88c6a7d\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:33.847759 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.847741    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ctbbd\" (UniqueName: \"kubernetes.io/projected/99777806-5fca-4857-96cc-db7dcb93da69-kube-api-access-ctbbd\") pod \"klusterlet-addon-workmgr-766d9794c8-l6z74\" (UID: \"99777806-5fca-4857-96cc-db7dcb93da69\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:33.911689 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911656    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-config\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.911868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-service-ca\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.911868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911723    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srpqp\" (UniqueName: \"kubernetes.io/projected/ada802fe-ca8b-4f1d-a8aa-920a67908065-kube-api-access-srpqp\") pod \"downloads-68c9cbb497-npb4s\" (UID: \"ada802fe-ca8b-4f1d-a8aa-920a67908065\") " pod="openshift-console/downloads-68c9cbb497-npb4s"
Jul 09 18:54:33.911868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911749    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-serving-cert\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.911868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911775    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pj2gd\" (UniqueName: \"kubernetes.io/projected/572f9a0d-e008-40a3-8f92-6f989eebdb8f-kube-api-access-pj2gd\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.911868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911794    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/407c46f4-31ca-4f90-a108-a2deab7ad827-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zxx77\" (UID: \"407c46f4-31ca-4f90-a108-a2deab7ad827\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"
Jul 09 18:54:33.911868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911824    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-oauth-config\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.911868 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.911858    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-oauth-serving-cert\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.912502 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.912462    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-config\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.912647 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.912551    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-service-ca\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.912647 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.912608    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-oauth-serving-cert\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.914356 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.914326    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/407c46f4-31ca-4f90-a108-a2deab7ad827-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-zxx77\" (UID: \"407c46f4-31ca-4f90-a108-a2deab7ad827\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"
Jul 09 18:54:33.914455 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.914369    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-oauth-config\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.914455 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.914372    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-serving-cert\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.946480 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.946419    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srpqp\" (UniqueName: \"kubernetes.io/projected/ada802fe-ca8b-4f1d-a8aa-920a67908065-kube-api-access-srpqp\") pod \"downloads-68c9cbb497-npb4s\" (UID: \"ada802fe-ca8b-4f1d-a8aa-920a67908065\") " pod="openshift-console/downloads-68c9cbb497-npb4s"
Jul 09 18:54:33.956348 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.956326    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pj2gd\" (UniqueName: \"kubernetes.io/projected/572f9a0d-e008-40a3-8f92-6f989eebdb8f-kube-api-access-pj2gd\") pod \"console-579df8cdd4-7r7z5\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") " pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:33.986089 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.986060    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:33.991944 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:33.991919    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:34.007316 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.007291    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"
Jul 09 18:54:34.013130 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.012969    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"
Jul 09 18:54:34.097775 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.097649    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"
Jul 09 18:54:34.105263 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.105211    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-68c9cbb497-npb4s"
Jul 09 18:54:34.111483 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.111277    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:34.212617 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.212499    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"]
Jul 09 18:54:34.215780 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:34.215732    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99777806_5fca_4857_96cc_db7dcb93da69.slice/crio-91775474462ac345d2ab447945692c67d9fcccc92f2632cb5bc983538e8fcdfa WatchSource:0}: Error finding container 91775474462ac345d2ab447945692c67d9fcccc92f2632cb5bc983538e8fcdfa: Status 404 returned error can't find the container with id 91775474462ac345d2ab447945692c67d9fcccc92f2632cb5bc983538e8fcdfa
Jul 09 18:54:34.243298 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.243256    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5"]
Jul 09 18:54:34.244925 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.244900    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6"]
Jul 09 18:54:34.248185 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.248127    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:34.252391 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:34.252346    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod93f41dd7_66bb_4673_9917_beddc88c6a7d.slice/crio-4651225cf89c883fa40d591d6ab7988e0671b58e78a7dd270497bcbac005d564 WatchSource:0}: Error finding container 4651225cf89c883fa40d591d6ab7988e0671b58e78a7dd270497bcbac005d564: Status 404 returned error can't find the container with id 4651225cf89c883fa40d591d6ab7988e0671b58e78a7dd270497bcbac005d564
Jul 09 18:54:34.253023 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:34.252989    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d81ff97_5313_4df1_ac0a_6050815722e3.slice/crio-c4f85f274398851bc1660cf811ee226d8953dc020219998a6b336d035be6e057 WatchSource:0}: Error finding container c4f85f274398851bc1660cf811ee226d8953dc020219998a6b336d035be6e057: Status 404 returned error can't find the container with id c4f85f274398851bc1660cf811ee226d8953dc020219998a6b336d035be6e057
Jul 09 18:54:34.313956 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.313927    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"]
Jul 09 18:54:34.358804 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.358774    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-68c9cbb497-npb4s"]
Jul 09 18:54:34.363642 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:34.363616    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod407c46f4_31ca_4f90_a108_a2deab7ad827.slice/crio-ad1dfa341fdf59dd758ddcec749eedcfea2828976ae5bd66008f3ab17a3232f4 WatchSource:0}: Error finding container ad1dfa341fdf59dd758ddcec749eedcfea2828976ae5bd66008f3ab17a3232f4: Status 404 returned error can't find the container with id ad1dfa341fdf59dd758ddcec749eedcfea2828976ae5bd66008f3ab17a3232f4
Jul 09 18:54:34.364362 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:34.364224    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podada802fe_ca8b_4f1d_a8aa_920a67908065.slice/crio-971d6151157eb4a38d7436f01897faaaf09e42fe4a284753eeda221c321e71e1 WatchSource:0}: Error finding container 971d6151157eb4a38d7436f01897faaaf09e42fe4a284753eeda221c321e71e1: Status 404 returned error can't find the container with id 971d6151157eb4a38d7436f01897faaaf09e42fe4a284753eeda221c321e71e1
Jul 09 18:54:34.388610 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.388582    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-579df8cdd4-7r7z5"]
Jul 09 18:54:34.391329 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:34.391288    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod572f9a0d_e008_40a3_8f92_6f989eebdb8f.slice/crio-f031b28188b3c6d17be230f0d4810aadad62f8d1de8ea5590d7b2826d0bd8a35 WatchSource:0}: Error finding container f031b28188b3c6d17be230f0d4810aadad62f8d1de8ea5590d7b2826d0bd8a35: Status 404 returned error can't find the container with id f031b28188b3c6d17be230f0d4810aadad62f8d1de8ea5590d7b2826d0bd8a35
Jul 09 18:54:34.464274 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.464179    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:34.676696 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.676661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77" event={"ID":"407c46f4-31ca-4f90-a108-a2deab7ad827","Type":"ContainerStarted","Data":"ad1dfa341fdf59dd758ddcec749eedcfea2828976ae5bd66008f3ab17a3232f4"}
Jul 09 18:54:34.677773 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.677746    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6" event={"ID":"93f41dd7-66bb-4673-9917-beddc88c6a7d","Type":"ContainerStarted","Data":"4651225cf89c883fa40d591d6ab7988e0671b58e78a7dd270497bcbac005d564"}
Jul 09 18:54:34.678772 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.678750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5" event={"ID":"0d81ff97-5313-4df1-ac0a-6050815722e3","Type":"ContainerStarted","Data":"c4f85f274398851bc1660cf811ee226d8953dc020219998a6b336d035be6e057"}
Jul 09 18:54:34.679746 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.679725    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74" event={"ID":"99777806-5fca-4857-96cc-db7dcb93da69","Type":"ContainerStarted","Data":"91775474462ac345d2ab447945692c67d9fcccc92f2632cb5bc983538e8fcdfa"}
Jul 09 18:54:34.680844 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.680821    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579df8cdd4-7r7z5" event={"ID":"572f9a0d-e008-40a3-8f92-6f989eebdb8f","Type":"ContainerStarted","Data":"f031b28188b3c6d17be230f0d4810aadad62f8d1de8ea5590d7b2826d0bd8a35"}
Jul 09 18:54:34.681827 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:34.681807    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-npb4s" event={"ID":"ada802fe-ca8b-4f1d-a8aa-920a67908065","Type":"ContainerStarted","Data":"971d6151157eb4a38d7436f01897faaaf09e42fe4a284753eeda221c321e71e1"}
Jul 09 18:54:36.701594 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:36.700683    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77" event={"ID":"407c46f4-31ca-4f90-a108-a2deab7ad827","Type":"ContainerStarted","Data":"69373397e4d2ca49d0c50437231b52404d67f41ee5cd8a9632d76d4e33865182"}
Jul 09 18:54:36.702151 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:36.702063    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"
Jul 09 18:54:36.709901 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:36.709850    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77"
Jul 09 18:54:36.719682 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:36.719627    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-zxx77" podStartSLOduration=1.610538043 podStartE2EDuration="3.719611224s" totalImagesPullingTime="2.109073181s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:36.717486047 +0000 UTC m=+87.069921125" watchObservedRunningTime="2026-07-09 18:54:36.719611224 +0000 UTC m=+87.072046271"
Jul 09 18:54:37.480300 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.480270    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:37.571909 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.570184    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-rsgtq"]
Jul 09 18:54:37.578960 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.578927    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:37.579623 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.579363    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.586986 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.586963    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 09 18:54:37.589447 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.589078    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-rsgtq"]
Jul 09 18:54:37.589447 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.589218    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 09 18:54:37.589447 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.589308    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-qlr92\""
Jul 09 18:54:37.590135 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.589964    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 09 18:54:37.653553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.653116    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-5knnr"
Jul 09 18:54:37.660867 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.660557    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jkz2c\" (UniqueName: \"kubernetes.io/projected/7234fe0c-4489-41b3-8868-7de2232675f7-kube-api-access-jkz2c\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.660867 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.660608    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7234fe0c-4489-41b3-8868-7de2232675f7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.660867 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.660677    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7234fe0c-4489-41b3-8868-7de2232675f7-metrics-client-ca\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.660867 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.660732    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7234fe0c-4489-41b3-8868-7de2232675f7-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.734912 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.734798    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:37.765402 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.764020    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jkz2c\" (UniqueName: \"kubernetes.io/projected/7234fe0c-4489-41b3-8868-7de2232675f7-kube-api-access-jkz2c\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.765402 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.764073    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7234fe0c-4489-41b3-8868-7de2232675f7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.765402 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.764185    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7234fe0c-4489-41b3-8868-7de2232675f7-metrics-client-ca\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.765402 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.764259    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7234fe0c-4489-41b3-8868-7de2232675f7-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.767136 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.766429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/7234fe0c-4489-41b3-8868-7de2232675f7-metrics-client-ca\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.769184 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.769139    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/7234fe0c-4489-41b3-8868-7de2232675f7-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.775765 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.775715    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/7234fe0c-4489-41b3-8868-7de2232675f7-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.778285 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.778243    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jkz2c\" (UniqueName: \"kubernetes.io/projected/7234fe0c-4489-41b3-8868-7de2232675f7-kube-api-access-jkz2c\") pod \"prometheus-operator-64785b6675-rsgtq\" (UID: \"7234fe0c-4489-41b3-8868-7de2232675f7\") " pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.882264 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.882157    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:37.903982 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.903737    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq"
Jul 09 18:54:37.953068 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:37.953011    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:38.684250 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:38.684199    2556 patch_prober.go:28] interesting pod/image-registry-7c4b4f9d7-n2cgw container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 09 18:54:38.684453 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:38.684257    2556 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw" podUID="0063f3df-5e0f-4576-8348-81aa821a2857" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:54:38.795299 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:38.795192    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:38.874343 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:38.874313    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:39.056107 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:39.056010    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:39.154742 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:39.154704    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:39.624319 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:39.624225    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:39.722760 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:39.722725    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:41.148022 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:41.147983    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:41.207317 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:41.207264    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:42.626766 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:42.626665    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:54:42.907493 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:42.907052    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:43.008156 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.008078    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:43.050155 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.050123    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-rsgtq"]
Jul 09 18:54:43.053040 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:43.053007    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7234fe0c_4489_41b3_8868_7de2232675f7.slice/crio-fadb8cc68a67ca6af2713bab6f892a28679436325a752ee60e0e080e0cc6a08f WatchSource:0}: Error finding container fadb8cc68a67ca6af2713bab6f892a28679436325a752ee60e0e080e0cc6a08f: Status 404 returned error can't find the container with id fadb8cc68a67ca6af2713bab6f892a28679436325a752ee60e0e080e0cc6a08f
Jul 09 18:54:43.741144 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.740854    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6" event={"ID":"93f41dd7-66bb-4673-9917-beddc88c6a7d","Type":"ContainerStarted","Data":"7b992b79923e22312d6d379b3684654101f514cdcf3a15a8aac2784b8d0b11a9"}
Jul 09 18:54:43.743882 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.743730    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5" event={"ID":"0d81ff97-5313-4df1-ac0a-6050815722e3","Type":"ContainerStarted","Data":"a0a846160573f38b01cb47a9577aa99a483fcdeb51e16b0ff8e1de5f6be5d5b9"}
Jul 09 18:54:43.743882 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.743772    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5" event={"ID":"0d81ff97-5313-4df1-ac0a-6050815722e3","Type":"ContainerStarted","Data":"66bc60794554d768ce9739016a16b955cc583a3b17621cfe0c66d62b9821ab66"}
Jul 09 18:54:43.743882 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.743790    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5" event={"ID":"0d81ff97-5313-4df1-ac0a-6050815722e3","Type":"ContainerStarted","Data":"355bedb8b0e7d9845fd501a0ad384c1a4ce54e08954f68a35997335c78098a6c"}
Jul 09 18:54:43.745688 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.745649    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74" event={"ID":"99777806-5fca-4857-96cc-db7dcb93da69","Type":"ContainerStarted","Data":"36b59c0a94d40a6cac38def87b0aeead20273de35590289ef286584a938e19f0"}
Jul 09 18:54:43.745975 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.745920    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:43.747808 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.747304    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq" event={"ID":"7234fe0c-4489-41b3-8868-7de2232675f7","Type":"ContainerStarted","Data":"fadb8cc68a67ca6af2713bab6f892a28679436325a752ee60e0e080e0cc6a08f"}
Jul 09 18:54:43.749032 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.749008    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74"
Jul 09 18:54:43.750891 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.750866    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579df8cdd4-7r7z5" event={"ID":"572f9a0d-e008-40a3-8f92-6f989eebdb8f","Type":"ContainerStarted","Data":"17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d"}
Jul 09 18:54:43.758071 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.758022    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-d8fcc5bf8-kwxh6" podStartSLOduration=2.1557305270000002 podStartE2EDuration="10.758005511s" totalImagesPullingTime="8.602274984s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:43.757267872 +0000 UTC m=+94.109702941" watchObservedRunningTime="2026-07-09 18:54:43.758005511 +0000 UTC m=+94.110440577"
Jul 09 18:54:43.777682 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.777636    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-766d9794c8-l6z74" podStartSLOduration=2.119720987 podStartE2EDuration="10.777608151s" totalImagesPullingTime="8.657887164s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:43.777374242 +0000 UTC m=+94.129809308" watchObservedRunningTime="2026-07-09 18:54:43.777608151 +0000 UTC m=+94.130043215"
Jul 09 18:54:43.798453 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.798405    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-579df8cdd4-7r7z5" podStartSLOduration=2.317038664 podStartE2EDuration="10.798387621s" totalImagesPullingTime="8.481348957s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:43.79744964 +0000 UTC m=+94.149884711" watchObservedRunningTime="2026-07-09 18:54:43.798387621 +0000 UTC m=+94.150822687"
Jul 09 18:54:43.815488 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:43.815448    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-5fcff995b-6bvr5" podStartSLOduration=2.188079221 podStartE2EDuration="10.815435549s" totalImagesPullingTime="8.627356328s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:43.814995407 +0000 UTC m=+94.167430472" watchObservedRunningTime="2026-07-09 18:54:43.815435549 +0000 UTC m=+94.167870613"
Jul 09 18:54:44.112110 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.112015    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:44.112110 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.112073    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:44.118309 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.118272    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:44.713026 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.712936    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:44.757199 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.757161    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq" event={"ID":"7234fe0c-4489-41b3-8868-7de2232675f7","Type":"ContainerStarted","Data":"8ffe440f42306d104151cf0dddcaf7f3b34a38b48b58052399ffaae8b64f2231"}
Jul 09 18:54:44.757586 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.757241    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq" event={"ID":"7234fe0c-4489-41b3-8868-7de2232675f7","Type":"ContainerStarted","Data":"b15b9efb3aea9708b6fd371c6e1273c37240cf42499b72b80736c7c6b56adcc9"}
Jul 09 18:54:44.760992 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.760972    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:54:44.778584 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.778542    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-rsgtq" podStartSLOduration=6.413316974 podStartE2EDuration="7.778515325s" totalImagesPullingTime="1.365198351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:44.77628605 +0000 UTC m=+95.128721117" watchObservedRunningTime="2026-07-09 18:54:44.778515325 +0000 UTC m=+95.130950391"
Jul 09 18:54:44.873480 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:44.873440    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:45.453884 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:45.453846    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:45.556831 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:45.556791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:46.076702 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.076666    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:46.178063 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.178030    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:46.679111 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.679078    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:46.812889 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.812844    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:46.915087 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.915049    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:46.971954 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.971739    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-95qtj"]
Jul 09 18:54:46.976003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.975982    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:46.978865 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.978843    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-7vtbj\""
Jul 09 18:54:46.978965 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.978913    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 09 18:54:46.979134 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.979117    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 09 18:54:46.979183 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:46.979119    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 09 18:54:47.017175 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.017146    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.056957    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85eb96a2-9d58-4ff3-b18e-58e30877ff63-metrics-client-ca\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057058    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057143    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-accelerators-collector-config\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057176    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-textfile\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057202    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bp7ts\" (UniqueName: \"kubernetes.io/projected/85eb96a2-9d58-4ff3-b18e-58e30877ff63-kube-api-access-bp7ts\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057292    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-tls\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057337    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-root\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057403    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-wtmp\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.057522 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.057472    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-sys\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.124751 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.124709    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:47.158819 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.158755    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-tls\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.158819 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.158809    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-root\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.159070 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.158849    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-wtmp\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.159070 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.158889    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-sys\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.159070 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.158952    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85eb96a2-9d58-4ff3-b18e-58e30877ff63-metrics-client-ca\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.159300 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.159282    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-sys\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.159358 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.159045    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-root\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.159693 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.159661    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-wtmp\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.159949 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.159927    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.160012 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.159999    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-accelerators-collector-config\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.160065 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.160035    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-textfile\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.160114 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.160081    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bp7ts\" (UniqueName: \"kubernetes.io/projected/85eb96a2-9d58-4ff3-b18e-58e30877ff63-kube-api-access-bp7ts\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.160782 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.160742    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-textfile\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.160976 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.160954    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/85eb96a2-9d58-4ff3-b18e-58e30877ff63-metrics-client-ca\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.161127 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.161106    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-accelerators-collector-config\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.162902 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.162877    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-tls\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.163488 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.163441    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/85eb96a2-9d58-4ff3-b18e-58e30877ff63-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.169632 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.169589    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bp7ts\" (UniqueName: \"kubernetes.io/projected/85eb96a2-9d58-4ff3-b18e-58e30877ff63-kube-api-access-bp7ts\") pod \"node-exporter-95qtj\" (UID: \"85eb96a2-9d58-4ff3-b18e-58e30877ff63\") " pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.235988 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.235855    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:47.287590 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.287560    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-95qtj"
Jul 09 18:54:47.306656 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:54:47.306618    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85eb96a2_9d58_4ff3_b18e_58e30877ff63.slice/crio-df41abe036cd0ea1a8abadc28ada02dd15e6c09581af5f76aa301f75543a8fec WatchSource:0}: Error finding container df41abe036cd0ea1a8abadc28ada02dd15e6c09581af5f76aa301f75543a8fec: Status 404 returned error can't find the container with id df41abe036cd0ea1a8abadc28ada02dd15e6c09581af5f76aa301f75543a8fec
Jul 09 18:54:47.326342 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.326250    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:47.440400 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.440337    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:47.570998 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.570909    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-rs466"
Jul 09 18:54:47.623773 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.623743    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:47.725269 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.725239    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:47.780539 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:47.780488    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95qtj" event={"ID":"85eb96a2-9d58-4ff3-b18e-58e30877ff63","Type":"ContainerStarted","Data":"df41abe036cd0ea1a8abadc28ada02dd15e6c09581af5f76aa301f75543a8fec"}
Jul 09 18:54:48.008321 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:48.008282    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:48.118512 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:48.117682    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:48.204924 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:48.204785    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:48.512421 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:48.512393    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:48.751271 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:48.751230    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:49.893657 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:49.893616    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:49.993039 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:49.992995    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:50.355197 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:50.355154    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:50.436193 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:50.436158    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:50.518586 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:50.518487    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:51.758262 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:51.758224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:51.857943 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:51.857900    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:51.965435 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:51.965398    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:52.164197 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:52.164116    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:52.248888 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:52.248851    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:52.493170 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:52.493131    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:52.566854 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:52.566814    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:53.216665 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:53.216622    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:53.504516 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:53.504425    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:53.653520 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:53.653442    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:57.646547 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.646489    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:57.748163 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.748125    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:57.830085 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.830052    2556 generic.go:356] "Generic (PLEG): container finished" podID="85eb96a2-9d58-4ff3-b18e-58e30877ff63" containerID="f9e55d5cb6b92f880db6bf7d8a6be4b7d648a8a8b616a7bf4fd24a7d713bc91e" exitCode=0
Jul 09 18:54:57.830388 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.830197    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95qtj" event={"ID":"85eb96a2-9d58-4ff3-b18e-58e30877ff63","Type":"ContainerDied","Data":"f9e55d5cb6b92f880db6bf7d8a6be4b7d648a8a8b616a7bf4fd24a7d713bc91e"}
Jul 09 18:54:57.832687 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.832661    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-68c9cbb497-npb4s" event={"ID":"ada802fe-ca8b-4f1d-a8aa-920a67908065","Type":"ContainerStarted","Data":"c18bda627cf58bed00fc4ce20573afbfe160bbb5d19f9bda51c99ff2238eeb0b"}
Jul 09 18:54:57.833377 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.833351    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-68c9cbb497-npb4s"
Jul 09 18:54:57.850304 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.850286    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-68c9cbb497-npb4s"
Jul 09 18:54:57.886230 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:57.886183    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-68c9cbb497-npb4s" podStartSLOduration=2.345373 podStartE2EDuration="24.886166114s" totalImagesPullingTime="22.540793114s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:54:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:57.885802902 +0000 UTC m=+108.238237969" watchObservedRunningTime="2026-07-09 18:54:57.886166114 +0000 UTC m=+108.238601179"
Jul 09 18:54:58.513808 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:58.513767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:58.612047 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:58.612008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:58.840623 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:58.840511    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95qtj" event={"ID":"85eb96a2-9d58-4ff3-b18e-58e30877ff63","Type":"ContainerStarted","Data":"fc2243cf2984ceedda340d73406e5192ebdc8c0cc0dda4e3e9c39875ae0a0bb2"}
Jul 09 18:54:58.840623 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:58.840593    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-95qtj" event={"ID":"85eb96a2-9d58-4ff3-b18e-58e30877ff63","Type":"ContainerStarted","Data":"334893de63302c41fda070f05670eb75438d14e0f72cf75d130459b738a7516e"}
Jul 09 18:54:58.861827 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:58.861746    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-95qtj" podStartSLOduration=3.222286692 podStartE2EDuration="12.861726982s" totalImagesPullingTime="9.551426426s" totalInitContainerRuntime="88.013864ms" isStatefulPod=true podCreationTimestamp="2026-07-09 18:54:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:54:58.860548201 +0000 UTC m=+109.212983266" watchObservedRunningTime="2026-07-09 18:54:58.861726982 +0000 UTC m=+109.214162049"
Jul 09 18:54:59.520440 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:59.520406    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:54:59.601894 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:54:59.601791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:02.011596 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:02.011560    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"]
Jul 09 18:55:02.097447 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:02.097404    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:02.197855 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:02.197812    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:02.301959 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:02.301872    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:02.373255 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:02.373220    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:03.871012 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:03.870977    2556 generic.go:356] "Generic (PLEG): container finished" podID="ff7c7467-a4b1-44aa-8dd8-50ed702d7f87" containerID="ccabb6dfb32860fec0bb901ba87025f5ad66341202f80aa3469a0717bd18917c" exitCode=0
Jul 09 18:55:03.871520 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:03.871038    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz" event={"ID":"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87","Type":"ContainerDied","Data":"ccabb6dfb32860fec0bb901ba87025f5ad66341202f80aa3469a0717bd18917c"}
Jul 09 18:55:03.871520 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:03.871443    2556 scope.go:117] "RemoveContainer" containerID="ccabb6dfb32860fec0bb901ba87025f5ad66341202f80aa3469a0717bd18917c"
Jul 09 18:55:04.532158 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:04.532126    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:04.633679 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:04.633645    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:04.878412 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:04.878336    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator-operator/kube-storage-version-migrator-operator-799cb7f958-rgrvz" event={"ID":"ff7c7467-a4b1-44aa-8dd8-50ed702d7f87","Type":"ContainerStarted","Data":"b68fc379f8f08651ee73ed6ad920ede443d1a224e3aec86481140de9ba6a6dea"}
Jul 09 18:55:07.892232 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:07.892199    2556 generic.go:356] "Generic (PLEG): container finished" podID="bfe48b74-fb4f-4e95-b58d-c05fb602c43c" containerID="2fe0c35ee1471966ac56d38a074549b63420fc2e233d74e60df807164322590b" exitCode=0
Jul 09 18:55:07.892710 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:07.892283    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz" event={"ID":"bfe48b74-fb4f-4e95-b58d-c05fb602c43c","Type":"ContainerDied","Data":"2fe0c35ee1471966ac56d38a074549b63420fc2e233d74e60df807164322590b"}
Jul 09 18:55:07.892780 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:07.892765    2556 scope.go:117] "RemoveContainer" containerID="2fe0c35ee1471966ac56d38a074549b63420fc2e233d74e60df807164322590b"
Jul 09 18:55:08.899326 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:08.899294    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-service-ca-operator/service-ca-operator-7695fc6c8c-qmxxz" event={"ID":"bfe48b74-fb4f-4e95-b58d-c05fb602c43c","Type":"ContainerStarted","Data":"ea97afaf4842f2e60c80dfcebddbc90ff0b7f67148dba8f728787a3dce289478"}
Jul 09 18:55:09.166288 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:09.166207    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579df8cdd4-7r7z5"]
Jul 09 18:55:09.268960 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:09.268923    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:09.375252 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:09.375219    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:09.475856 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:09.475821    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:13.922302 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:13.922270    2556 generic.go:356] "Generic (PLEG): container finished" podID="569b9f3d-0fa3-425a-9815-a374ea2780a6" containerID="ac885175eaed1f7944b6b4b5030dd68ccbcfa3af32038f00bcc4f8eb2cf63a60" exitCode=0
Jul 09 18:55:13.922673 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:13.922354    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x" event={"ID":"569b9f3d-0fa3-425a-9815-a374ea2780a6","Type":"ContainerDied","Data":"ac885175eaed1f7944b6b4b5030dd68ccbcfa3af32038f00bcc4f8eb2cf63a60"}
Jul 09 18:55:13.922727 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:13.922715    2556 scope.go:117] "RemoveContainer" containerID="ac885175eaed1f7944b6b4b5030dd68ccbcfa3af32038f00bcc4f8eb2cf63a60"
Jul 09 18:55:14.928445 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:14.928412    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-operator-68d9dd94cf-7zc7x" event={"ID":"569b9f3d-0fa3-425a-9815-a374ea2780a6","Type":"ContainerStarted","Data":"f2a0e42bbe3bb34b718af8865bfc0eae6f7a764bab70e24441f4f78149574a48"}
Jul 09 18:55:15.064754 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:15.064722    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66ddf58c96-f6w75_7d5933ef-2952-4201-891b-78f29352a829/router/0.log"
Jul 09 18:55:15.070870 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:15.070839    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ltkwk_b49c8ad2-d3d4-44bd-91d1-d454284d0c64/serve-healthcheck-canary/0.log"
Jul 09 18:55:27.036688 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.036647    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw" podUID="0063f3df-5e0f-4576-8348-81aa821a2857" containerName="registry" containerID="cri-o://aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823" gracePeriod=30
Jul 09 18:55:27.314725 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.314699    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.443819    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-trusted-ca\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.443876    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.443947    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0063f3df-5e0f-4576-8348-81aa821a2857-ca-trust-extracted\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.443992    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-image-registry-private-configuration\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.444025    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-registry-certificates\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.444067    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-bound-sa-token\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.444091    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xjqwb\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-kube-api-access-xjqwb\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.444881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.444127    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-installation-pull-secrets\") pod \"0063f3df-5e0f-4576-8348-81aa821a2857\" (UID: \"0063f3df-5e0f-4576-8348-81aa821a2857\") "
Jul 09 18:55:27.445432 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.444950    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:55:27.445866 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.445838    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:55:27.447897 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.447873    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:55:27.449448 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.449408    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:55:27.449589 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.449435    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:55:27.449954 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.449924    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:55:27.450067 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.449982    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-kube-api-access-xjqwb" (OuterVolumeSpecName: "kube-api-access-xjqwb") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "kube-api-access-xjqwb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:55:27.455331 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.455302    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0063f3df-5e0f-4576-8348-81aa821a2857-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "0063f3df-5e0f-4576-8348-81aa821a2857" (UID: "0063f3df-5e0f-4576-8348-81aa821a2857"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545012    2556 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-bound-sa-token\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545044    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xjqwb\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-kube-api-access-xjqwb\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545062    2556 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-installation-pull-secrets\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545076    2556 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-trusted-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545089    2556 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0063f3df-5e0f-4576-8348-81aa821a2857-registry-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545102    2556 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0063f3df-5e0f-4576-8348-81aa821a2857-ca-trust-extracted\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545118    2556 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0063f3df-5e0f-4576-8348-81aa821a2857-image-registry-private-configuration\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.545226 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.545131    2556 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0063f3df-5e0f-4576-8348-81aa821a2857-registry-certificates\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:27.988279 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.988250    2556 generic.go:356] "Generic (PLEG): container finished" podID="0063f3df-5e0f-4576-8348-81aa821a2857" containerID="aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823" exitCode=0
Jul 09 18:55:27.988457 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.988290    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"
Jul 09 18:55:27.988457 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.988309    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw" event={"ID":"0063f3df-5e0f-4576-8348-81aa821a2857","Type":"ContainerDied","Data":"aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823"}
Jul 09 18:55:27.988457 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.988348    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7c4b4f9d7-n2cgw" event={"ID":"0063f3df-5e0f-4576-8348-81aa821a2857","Type":"ContainerDied","Data":"a07a350fd830c60a791e478c675877f6ceb3d0c71f0c3e494f0ffc44136a4fe0"}
Jul 09 18:55:27.988457 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:27.988367    2556 scope.go:117] "RemoveContainer" containerID="aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823"
Jul 09 18:55:28.002932 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:28.002903    2556 scope.go:117] "RemoveContainer" containerID="aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823"
Jul 09 18:55:28.003894 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:55:28.003862    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823\": container with ID starting with aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823 not found: ID does not exist" containerID="aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823"
Jul 09 18:55:28.004292 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:28.004254    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823"} err="failed to get container status \"aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823\": rpc error: code = NotFound desc = could not find container \"aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823\": container with ID starting with aa6186c0b97cd3a4bd773318a9c5b38a3ebf63ec8f6f45c9f441ffcb0336f823 not found: ID does not exist"
Jul 09 18:55:28.022916 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:28.022884    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"]
Jul 09 18:55:28.027821 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:28.027788    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7c4b4f9d7-n2cgw"]
Jul 09 18:55:28.085003 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:28.084972    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0063f3df-5e0f-4576-8348-81aa821a2857" path="/var/lib/kubelet/pods/0063f3df-5e0f-4576-8348-81aa821a2857/volumes"
Jul 09 18:55:28.500588 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:28.500549    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:28.595171 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:28.595136    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:33.553085 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:33.553046    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:33.652635 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:33.652598    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:33.710279 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:33.710233    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:55:34.190197 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.190138    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-579df8cdd4-7r7z5" podUID="572f9a0d-e008-40a3-8f92-6f989eebdb8f" containerName="console" containerID="cri-o://17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d" gracePeriod=15
Jul 09 18:55:34.473733 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.473671    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:55:34.613266 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613236    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-config\") pod \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") "
Jul 09 18:55:34.613708 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613274    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pj2gd\" (UniqueName: \"kubernetes.io/projected/572f9a0d-e008-40a3-8f92-6f989eebdb8f-kube-api-access-pj2gd\") pod \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") "
Jul 09 18:55:34.613708 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613299    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-oauth-serving-cert\") pod \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") "
Jul 09 18:55:34.613708 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613346    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-service-ca\") pod \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") "
Jul 09 18:55:34.613708 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613375    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-serving-cert\") pod \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") "
Jul 09 18:55:34.613708 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613427    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-oauth-config\") pod \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\" (UID: \"572f9a0d-e008-40a3-8f92-6f989eebdb8f\") "
Jul 09 18:55:34.613969 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613860    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-service-ca" (OuterVolumeSpecName: "service-ca") pod "572f9a0d-e008-40a3-8f92-6f989eebdb8f" (UID: "572f9a0d-e008-40a3-8f92-6f989eebdb8f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:55:34.613969 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613767    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "572f9a0d-e008-40a3-8f92-6f989eebdb8f" (UID: "572f9a0d-e008-40a3-8f92-6f989eebdb8f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:55:34.614050 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.613956    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-config" (OuterVolumeSpecName: "console-config") pod "572f9a0d-e008-40a3-8f92-6f989eebdb8f" (UID: "572f9a0d-e008-40a3-8f92-6f989eebdb8f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:55:34.615710 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.615679    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/572f9a0d-e008-40a3-8f92-6f989eebdb8f-kube-api-access-pj2gd" (OuterVolumeSpecName: "kube-api-access-pj2gd") pod "572f9a0d-e008-40a3-8f92-6f989eebdb8f" (UID: "572f9a0d-e008-40a3-8f92-6f989eebdb8f"). InnerVolumeSpecName "kube-api-access-pj2gd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:55:34.615807 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.615725    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "572f9a0d-e008-40a3-8f92-6f989eebdb8f" (UID: "572f9a0d-e008-40a3-8f92-6f989eebdb8f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:55:34.615807 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.615736    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "572f9a0d-e008-40a3-8f92-6f989eebdb8f" (UID: "572f9a0d-e008-40a3-8f92-6f989eebdb8f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:55:34.714302 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.714264    2556 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-serving-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:34.714302 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.714296    2556 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-oauth-config\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:34.714302 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.714306    2556 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-console-config\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:34.714519 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.714315    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pj2gd\" (UniqueName: \"kubernetes.io/projected/572f9a0d-e008-40a3-8f92-6f989eebdb8f-kube-api-access-pj2gd\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:34.714519 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.714325    2556 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-oauth-serving-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:34.714519 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:34.714333    2556 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/572f9a0d-e008-40a3-8f92-6f989eebdb8f-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:55:35.026545 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.026495    2556 generic.go:356] "Generic (PLEG): container finished" podID="572f9a0d-e008-40a3-8f92-6f989eebdb8f" containerID="17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d" exitCode=0
Jul 09 18:55:35.026716 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.026557    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-579df8cdd4-7r7z5"
Jul 09 18:55:35.026716 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.026572    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579df8cdd4-7r7z5" event={"ID":"572f9a0d-e008-40a3-8f92-6f989eebdb8f","Type":"ContainerDied","Data":"17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d"}
Jul 09 18:55:35.026716 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.026622    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-579df8cdd4-7r7z5" event={"ID":"572f9a0d-e008-40a3-8f92-6f989eebdb8f","Type":"ContainerDied","Data":"f031b28188b3c6d17be230f0d4810aadad62f8d1de8ea5590d7b2826d0bd8a35"}
Jul 09 18:55:35.026716 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.026647    2556 scope.go:117] "RemoveContainer" containerID="17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d"
Jul 09 18:55:35.035644 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.035629    2556 scope.go:117] "RemoveContainer" containerID="17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d"
Jul 09 18:55:35.035905 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:55:35.035886    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d\": container with ID starting with 17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d not found: ID does not exist" containerID="17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d"
Jul 09 18:55:35.035967 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.035912    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d"} err="failed to get container status \"17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d\": rpc error: code = NotFound desc = could not find container \"17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d\": container with ID starting with 17bbc21c18b5b7eeb4b2646b447c0863e5088bb571ee7f437a051f6aa195639d not found: ID does not exist"
Jul 09 18:55:35.048770 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.048745    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-579df8cdd4-7r7z5"]
Jul 09 18:55:35.052318 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:35.052299    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-579df8cdd4-7r7z5"]
Jul 09 18:55:36.083148 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:55:36.083117    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="572f9a0d-e008-40a3-8f92-6f989eebdb8f" path="/var/lib/kubelet/pods/572f9a0d-e008-40a3-8f92-6f989eebdb8f/volumes"
Jul 09 18:56:03.121820 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:03.121736    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:03.213884 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:03.213847    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:15.083513 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:15.083477    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:15.171885 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:15.171853    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:22.620607 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:22.620566    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:22.761936 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:22.761903    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:22.821750 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:22.821716    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:23.375460 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:23.375417    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:23.473001 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:23.472963    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:28.736243 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:28.736207    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:28.842279 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:28.842242    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:28.928928 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:28.928894    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:33.605907 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:33.605875    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:33.705811 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:33.705773    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:33.767595 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:33.767565    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:45.209806 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:45.209765    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:45.274994 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:45.274964    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:56:47.015256 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:56:47.015219    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:11.815543 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:11.815499    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:11.915335 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:11.915302    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:33.880957 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:33.880875    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:35.313662 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:35.313627    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:37.508453 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:37.508410    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:37.605216 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:37.605182    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:39.489798 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:39.489764    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:39.592032 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:39.592000    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:40.129978 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:40.129943    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:40.256560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:40.256490    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:41.371332 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:41.371295    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:41.986877 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:41.986836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:44.490025 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:44.489989    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:44.550640 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:44.550607    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:50.696376 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:50.696342    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:52.871691 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:52.871659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:52.970996 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:52.970964    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:58.885121 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:58.885076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:57:58.982581 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:57:58.982545    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:00.859630 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:00.859600    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:01.436624 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:01.436590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:07.033393 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:07.033349    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:07.134162 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:07.134124    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:09.985734 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:09.985705    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 18:58:09.986156 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:09.985742    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 18:58:09.991722 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:09.991700    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 18:58:09.991882 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:09.991701    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 18:58:09.995593 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:09.995571    2556 kubelet.go:1648] "Image garbage collection succeeded"
Jul 09 18:58:12.532115 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:12.532078    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:13.537099 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.537063    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"]
Jul 09 18:58:13.537467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.537409    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="572f9a0d-e008-40a3-8f92-6f989eebdb8f" containerName="console"
Jul 09 18:58:13.537467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.537422    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="572f9a0d-e008-40a3-8f92-6f989eebdb8f" containerName="console"
Jul 09 18:58:13.537467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.537445    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0063f3df-5e0f-4576-8348-81aa821a2857" containerName="registry"
Jul 09 18:58:13.537467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.537451    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="0063f3df-5e0f-4576-8348-81aa821a2857" containerName="registry"
Jul 09 18:58:13.537668 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.537508    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="0063f3df-5e0f-4576-8348-81aa821a2857" containerName="registry"
Jul 09 18:58:13.537668 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.537522    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="572f9a0d-e008-40a3-8f92-6f989eebdb8f" containerName="console"
Jul 09 18:58:13.539383 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.539366    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.542452 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.542430    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 09 18:58:13.542681 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.542668    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 09 18:58:13.542747 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.542687    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 09 18:58:13.542747 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.542710    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 09 18:58:13.543092 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.543075    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-xxtqd\""
Jul 09 18:58:13.563074 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.563042    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"]
Jul 09 18:58:13.588755 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.588722    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e24ffc53-a74f-4ee1-b917-bff6147474f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.588891 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.588762    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xk5vc\" (UniqueName: \"kubernetes.io/projected/e24ffc53-a74f-4ee1-b917-bff6147474f2-kube-api-access-xk5vc\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.588891 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.588784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e24ffc53-a74f-4ee1-b917-bff6147474f2-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.588891 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.588853    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24ffc53-a74f-4ee1-b917-bff6147474f2-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.588891 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.588881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e24ffc53-a74f-4ee1-b917-bff6147474f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.687081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.687048    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:13.690092 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.690069    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e24ffc53-a74f-4ee1-b917-bff6147474f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.690199 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.690099    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xk5vc\" (UniqueName: \"kubernetes.io/projected/e24ffc53-a74f-4ee1-b917-bff6147474f2-kube-api-access-xk5vc\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.690199 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.690118    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e24ffc53-a74f-4ee1-b917-bff6147474f2-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.690199 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.690160    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24ffc53-a74f-4ee1-b917-bff6147474f2-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.690199 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.690188    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e24ffc53-a74f-4ee1-b917-bff6147474f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.690633 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.690615    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/e24ffc53-a74f-4ee1-b917-bff6147474f2-manifests\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.692509 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.692486    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/e24ffc53-a74f-4ee1-b917-bff6147474f2-tmp\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.692617 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.692575    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/e24ffc53-a74f-4ee1-b917-bff6147474f2-webhook-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.692679 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.692644    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/e24ffc53-a74f-4ee1-b917-bff6147474f2-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.698307 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.698285    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xk5vc\" (UniqueName: \"kubernetes.io/projected/e24ffc53-a74f-4ee1-b917-bff6147474f2-kube-api-access-xk5vc\") pod \"opendatahub-operator-controller-manager-5cd98cb899-ddcnl\" (UID: \"e24ffc53-a74f-4ee1-b917-bff6147474f2\") " pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.785137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.785099    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:13.849862 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.849784    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:13.891834 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.891775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:13.989740 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.989711    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"]
Jul 09 18:58:13.996223 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:58:13.996198    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode24ffc53_a74f_4ee1_b917_bff6147474f2.slice/crio-e50f472c519185fe44bc5fabeed56323b70a0dcb140071f8219b0337b24b2e16 WatchSource:0}: Error finding container e50f472c519185fe44bc5fabeed56323b70a0dcb140071f8219b0337b24b2e16: Status 404 returned error can't find the container with id e50f472c519185fe44bc5fabeed56323b70a0dcb140071f8219b0337b24b2e16
Jul 09 18:58:13.998148 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:13.998127    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 18:58:14.004164 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:14.004139    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:14.090204 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:14.090150    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:14.675420 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:14.675382    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl" event={"ID":"e24ffc53-a74f-4ee1-b917-bff6147474f2","Type":"ContainerStarted","Data":"e50f472c519185fe44bc5fabeed56323b70a0dcb140071f8219b0337b24b2e16"}
Jul 09 18:58:17.691590 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:17.691553    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl" event={"ID":"e24ffc53-a74f-4ee1-b917-bff6147474f2","Type":"ContainerStarted","Data":"e3f9233c154c1e306bd6c8c623895534f01092255119502f07eb9f42a9b098a4"}
Jul 09 18:58:18.697968 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:18.697931    2556 generic.go:356] "Generic (PLEG): container finished" podID="e24ffc53-a74f-4ee1-b917-bff6147474f2" containerID="e3f9233c154c1e306bd6c8c623895534f01092255119502f07eb9f42a9b098a4" exitCode=0
Jul 09 18:58:18.698400 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:18.698014    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl" event={"ID":"e24ffc53-a74f-4ee1-b917-bff6147474f2","Type":"ContainerDied","Data":"e3f9233c154c1e306bd6c8c623895534f01092255119502f07eb9f42a9b098a4"}
Jul 09 18:58:19.704766 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:19.704726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl" event={"ID":"e24ffc53-a74f-4ee1-b917-bff6147474f2","Type":"ContainerStarted","Data":"d3ceecc951fd60d55649dad88d1e272f8d431e125420aa1781137c051f2f1ddb"}
Jul 09 18:58:19.705138 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:19.704879    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:19.728752 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:19.728710    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl" podStartSLOduration=2.31833572 podStartE2EDuration="6.728693357s" totalImagesPullingTime="3.319549921s" totalInitContainerRuntime="1.090807716s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:13 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:19.727912837 +0000 UTC m=+310.080347902" watchObservedRunningTime="2026-07-09 18:58:19.728693357 +0000 UTC m=+310.081128509"
Jul 09 18:58:29.367628 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:29.367590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:29.523258 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:29.523212    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:30.710991 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:30.710959    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5cd98cb899-ddcnl"
Jul 09 18:58:32.602040 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:32.602004    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:32.806259 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:32.806231    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:33.293434 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.293405    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:33.492856 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.492823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:33.579070 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.578982    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"]
Jul 09 18:58:33.582223 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.582198    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.592753 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.592731    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 09 18:58:33.592855 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.592734    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 09 18:58:33.592855 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.592802    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-56g44\""
Jul 09 18:58:33.611284 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.611256    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"]
Jul 09 18:58:33.661102 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.661076    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61ab7487-5540-4a64-aa38-ed6e7a449d5c-tmp\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.661273 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.661122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5zvl8\" (UniqueName: \"kubernetes.io/projected/61ab7487-5540-4a64-aa38-ed6e7a449d5c-kube-api-access-5zvl8\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.661273 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.661193    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ab7487-5540-4a64-aa38-ed6e7a449d5c-tls-certs\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.715467 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.715430    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:33.761723 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.761686    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61ab7487-5540-4a64-aa38-ed6e7a449d5c-tmp\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.761900 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.761756    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5zvl8\" (UniqueName: \"kubernetes.io/projected/61ab7487-5540-4a64-aa38-ed6e7a449d5c-kube-api-access-5zvl8\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.761900 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.761795    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ab7487-5540-4a64-aa38-ed6e7a449d5c-tls-certs\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.764020 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.763994    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/61ab7487-5540-4a64-aa38-ed6e7a449d5c-tmp\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.764240 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.764221    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/61ab7487-5540-4a64-aa38-ed6e7a449d5c-tls-certs\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.781104 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.781073    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5zvl8\" (UniqueName: \"kubernetes.io/projected/61ab7487-5540-4a64-aa38-ed6e7a449d5c-kube-api-access-5zvl8\") pod \"kube-auth-proxy-5b58cdb447-zg657\" (UID: \"61ab7487-5540-4a64-aa38-ed6e7a449d5c\") " pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.858851 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.858768    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j97g2"]
Jul 09 18:58:33.862255 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.862238    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:33.864695 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.864672    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 09 18:58:33.864992 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.864977    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-pdqng\""
Jul 09 18:58:33.873046 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.873024    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j97g2"]
Jul 09 18:58:33.891574 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.891552    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"
Jul 09 18:58:33.950006 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.949971    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:33.963356 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.963327    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-552s9\" (UniqueName: \"kubernetes.io/projected/885633a5-f210-4edc-b348-df292898fc5c-kube-api-access-552s9\") pod \"odh-model-controller-7998c67dc4-j97g2\" (UID: \"885633a5-f210-4edc-b348-df292898fc5c\") " pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:33.963487 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:33.963372    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/885633a5-f210-4edc-b348-df292898fc5c-cert\") pod \"odh-model-controller-7998c67dc4-j97g2\" (UID: \"885633a5-f210-4edc-b348-df292898fc5c\") " pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:34.044186 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.044156    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5b58cdb447-zg657"]
Jul 09 18:58:34.047366 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:58:34.047331    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61ab7487_5540_4a64_aa38_ed6e7a449d5c.slice/crio-010bebd9aab56ad0393b86d3093f4409f90eb3e3656b05982c891f8725852747 WatchSource:0}: Error finding container 010bebd9aab56ad0393b86d3093f4409f90eb3e3656b05982c891f8725852747: Status 404 returned error can't find the container with id 010bebd9aab56ad0393b86d3093f4409f90eb3e3656b05982c891f8725852747
Jul 09 18:58:34.050347 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.050322    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:34.064806 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.064780    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-552s9\" (UniqueName: \"kubernetes.io/projected/885633a5-f210-4edc-b348-df292898fc5c-kube-api-access-552s9\") pod \"odh-model-controller-7998c67dc4-j97g2\" (UID: \"885633a5-f210-4edc-b348-df292898fc5c\") " pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:34.064906 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.064818    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/885633a5-f210-4edc-b348-df292898fc5c-cert\") pod \"odh-model-controller-7998c67dc4-j97g2\" (UID: \"885633a5-f210-4edc-b348-df292898fc5c\") " pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:34.064967 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:58:34.064932    2556 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 09 18:58:34.065016 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:58:34.064995    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/885633a5-f210-4edc-b348-df292898fc5c-cert podName:885633a5-f210-4edc-b348-df292898fc5c nodeName:}" failed. No retries permitted until 2026-07-09 18:58:34.564974739 +0000 UTC m=+324.917409782 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/885633a5-f210-4edc-b348-df292898fc5c-cert") pod "odh-model-controller-7998c67dc4-j97g2" (UID: "885633a5-f210-4edc-b348-df292898fc5c") : secret "odh-model-controller-webhook-cert" not found
Jul 09 18:58:34.087539 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.087506    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-552s9\" (UniqueName: \"kubernetes.io/projected/885633a5-f210-4edc-b348-df292898fc5c-kube-api-access-552s9\") pod \"odh-model-controller-7998c67dc4-j97g2\" (UID: \"885633a5-f210-4edc-b348-df292898fc5c\") " pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:34.145775 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.145664    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:34.360343 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.360305    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:34.569788 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.569750    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/885633a5-f210-4edc-b348-df292898fc5c-cert\") pod \"odh-model-controller-7998c67dc4-j97g2\" (UID: \"885633a5-f210-4edc-b348-df292898fc5c\") " pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:34.572229 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.572202    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/885633a5-f210-4edc-b348-df292898fc5c-cert\") pod \"odh-model-controller-7998c67dc4-j97g2\" (UID: \"885633a5-f210-4edc-b348-df292898fc5c\") " pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:34.772873 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.772841    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:34.774229 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.774206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657" event={"ID":"61ab7487-5540-4a64-aa38-ed6e7a449d5c","Type":"ContainerStarted","Data":"010bebd9aab56ad0393b86d3093f4409f90eb3e3656b05982c891f8725852747"}
Jul 09 18:58:34.884714 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.884659    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:34.932797 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.932764    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p8rb6"]
Jul 09 18:58:34.936612 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.936583    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:34.941033 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.941006    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-m8d2f\""
Jul 09 18:58:34.941327 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.941303    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 09 18:58:34.948273 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.948248    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-j97g2"]
Jul 09 18:58:34.951172 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:58:34.951147    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod885633a5_f210_4edc_b348_df292898fc5c.slice/crio-77449f621f4e560320dc1ed54ccdcfd54ab23d475ee3e6535432fa61d1807736 WatchSource:0}: Error finding container 77449f621f4e560320dc1ed54ccdcfd54ab23d475ee3e6535432fa61d1807736: Status 404 returned error can't find the container with id 77449f621f4e560320dc1ed54ccdcfd54ab23d475ee3e6535432fa61d1807736
Jul 09 18:58:34.952585 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.952563    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p8rb6"]
Jul 09 18:58:34.976870 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.976837    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"]
Jul 09 18:58:34.980718 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.980690    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:34.993137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.992994    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 09 18:58:34.993137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.993006    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 09 18:58:34.993137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.993017    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 18:58:34.993137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.993011    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-6b79v\""
Jul 09 18:58:34.993137 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:34.993006    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 09 18:58:35.003088 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.003057    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"]
Jul 09 18:58:35.035979 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.035935    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:35.075827 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.075731    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72qrb\" (UniqueName: \"kubernetes.io/projected/2aeba00d-5159-4b98-ad8c-dc649da0f565-kube-api-access-72qrb\") pod \"model-serving-api-54d89bd5fc-p8rb6\" (UID: \"2aeba00d-5159-4b98-ad8c-dc649da0f565\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:35.075827 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.075772    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-token\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.076055 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.075900    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ffbb3e-ef70-47a4-bcbd-21199d473045-local-certs\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.076055 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.075945    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4tsh2\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-kube-api-access-4tsh2\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.076055 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.075982    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.076055 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.076005    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.076196 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.076077    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeba00d-5159-4b98-ad8c-dc649da0f565-tls-certs\") pod \"model-serving-api-54d89bd5fc-p8rb6\" (UID: \"2aeba00d-5159-4b98-ad8c-dc649da0f565\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:35.076196 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.076132    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.076196 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.076171    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-cacerts\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.177382 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.177344    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.177382 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.177385    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.177672 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.177419    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeba00d-5159-4b98-ad8c-dc649da0f565-tls-certs\") pod \"model-serving-api-54d89bd5fc-p8rb6\" (UID: \"2aeba00d-5159-4b98-ad8c-dc649da0f565\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:35.177672 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.177462    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.177672 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:58:35.177601    2556 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 09 18:58:35.177823 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:58:35.177687    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2aeba00d-5159-4b98-ad8c-dc649da0f565-tls-certs podName:2aeba00d-5159-4b98-ad8c-dc649da0f565 nodeName:}" failed. No retries permitted until 2026-07-09 18:58:35.677664651 +0000 UTC m=+326.030099716 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/2aeba00d-5159-4b98-ad8c-dc649da0f565-tls-certs") pod "model-serving-api-54d89bd5fc-p8rb6" (UID: "2aeba00d-5159-4b98-ad8c-dc649da0f565") : secret "model-serving-api-tls" not found
Jul 09 18:58:35.178641 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.178017    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-cacerts\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.178641 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.178116    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72qrb\" (UniqueName: \"kubernetes.io/projected/2aeba00d-5159-4b98-ad8c-dc649da0f565-kube-api-access-72qrb\") pod \"model-serving-api-54d89bd5fc-p8rb6\" (UID: \"2aeba00d-5159-4b98-ad8c-dc649da0f565\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:35.178641 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.178152    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-token\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.178641 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.178209    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ffbb3e-ef70-47a4-bcbd-21199d473045-local-certs\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.178641 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.178241    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4tsh2\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-kube-api-access-4tsh2\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.178641 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.178378    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.180465 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.180423    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-kubeconfig\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.180885 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.180863    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-cacerts\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.181182 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.181161    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.182114 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.182071    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ffbb3e-ef70-47a4-bcbd-21199d473045-local-certs\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.183074 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.182976    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:35.195939 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.195910    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4tsh2\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-kube-api-access-4tsh2\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.197075 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.197045    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-token\") pod \"istiod-openshift-gateway-7574459656-nj8mf\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") " pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.198256 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.198207    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72qrb\" (UniqueName: \"kubernetes.io/projected/2aeba00d-5159-4b98-ad8c-dc649da0f565-kube-api-access-72qrb\") pod \"model-serving-api-54d89bd5fc-p8rb6\" (UID: \"2aeba00d-5159-4b98-ad8c-dc649da0f565\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:35.293039 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.293007    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:35.373073 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.373039    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:35.447550 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.447441    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"]
Jul 09 18:58:35.451818 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:58:35.451792    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44ffbb3e_ef70_47a4_bcbd_21199d473045.slice/crio-6611bd52c71733e06b29d13b7009e0c22f39f4bb7fdaa9c320fcf99616705115 WatchSource:0}: Error finding container 6611bd52c71733e06b29d13b7009e0c22f39f4bb7fdaa9c320fcf99616705115: Status 404 returned error can't find the container with id 6611bd52c71733e06b29d13b7009e0c22f39f4bb7fdaa9c320fcf99616705115
Jul 09 18:58:35.683954 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.683913    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeba00d-5159-4b98-ad8c-dc649da0f565-tls-certs\") pod \"model-serving-api-54d89bd5fc-p8rb6\" (UID: \"2aeba00d-5159-4b98-ad8c-dc649da0f565\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:35.687120 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.687094    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2aeba00d-5159-4b98-ad8c-dc649da0f565-tls-certs\") pod \"model-serving-api-54d89bd5fc-p8rb6\" (UID: \"2aeba00d-5159-4b98-ad8c-dc649da0f565\") " pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:35.769089 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.769055    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:35.785999 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.785965    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf" event={"ID":"44ffbb3e-ef70-47a4-bcbd-21199d473045","Type":"ContainerStarted","Data":"6611bd52c71733e06b29d13b7009e0c22f39f4bb7fdaa9c320fcf99616705115"}
Jul 09 18:58:35.787442 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.787416    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j97g2" event={"ID":"885633a5-f210-4edc-b348-df292898fc5c","Type":"ContainerStarted","Data":"77449f621f4e560320dc1ed54ccdcfd54ab23d475ee3e6535432fa61d1807736"}
Jul 09 18:58:35.837692 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.837602    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:35.849566 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:35.849514    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:36.030810 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:36.030362    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-p8rb6"]
Jul 09 18:58:36.034639 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:58:36.034593    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2aeba00d_5159_4b98_ad8c_dc649da0f565.slice/crio-2e40277f53401873e5ff44cae49fb07a6eb09e8c5ced1b8c6767652d71c08750 WatchSource:0}: Error finding container 2e40277f53401873e5ff44cae49fb07a6eb09e8c5ced1b8c6767652d71c08750: Status 404 returned error can't find the container with id 2e40277f53401873e5ff44cae49fb07a6eb09e8c5ced1b8c6767652d71c08750
Jul 09 18:58:36.371335 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:36.371115    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:36.797685 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:36.797647    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6" event={"ID":"2aeba00d-5159-4b98-ad8c-dc649da0f565","Type":"ContainerStarted","Data":"2e40277f53401873e5ff44cae49fb07a6eb09e8c5ced1b8c6767652d71c08750"}
Jul 09 18:58:39.902885 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:39.902848    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 18:58:39.903224 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:39.902916    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 18:58:40.819443 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.819404    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-j97g2" event={"ID":"885633a5-f210-4edc-b348-df292898fc5c","Type":"ContainerStarted","Data":"c73857f95d5c4cd1783afb5023bd9e8b0afc52f7fd2894dc3d6c69e56ee7fff4"}
Jul 09 18:58:40.819682 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.819556    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:40.820939 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.820910    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6" event={"ID":"2aeba00d-5159-4b98-ad8c-dc649da0f565","Type":"ContainerStarted","Data":"40fcae2141c57c210936638bcdf82a20f0f2abc3b0751a5106b492ceaf5c02f6"}
Jul 09 18:58:40.821075 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.821022    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:58:40.822488 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.822466    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf" event={"ID":"44ffbb3e-ef70-47a4-bcbd-21199d473045","Type":"ContainerStarted","Data":"06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931"}
Jul 09 18:58:40.822599 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.822511    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:40.823901 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.823880    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657" event={"ID":"61ab7487-5540-4a64-aa38-ed6e7a449d5c","Type":"ContainerStarted","Data":"388fb99871714e74b7e090fa0a8137133e4cc05bac1588bc739d5ecd7cb021d8"}
Jul 09 18:58:40.824445 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.824421    2556 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7574459656-nj8mf container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 09 18:58:40.824520 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.824468    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf" podUID="44ffbb3e-ef70-47a4-bcbd-21199d473045" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 09 18:58:40.844736 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.844686    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-j97g2" podStartSLOduration=2.886344186 podStartE2EDuration="7.844669865s" totalImagesPullingTime="4.958325679s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:40.844241855 +0000 UTC m=+331.196676920" watchObservedRunningTime="2026-07-09 18:58:40.844669865 +0000 UTC m=+331.197104932"
Jul 09 18:58:40.864998 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.864942    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5b58cdb447-zg657" podStartSLOduration=2.015847156 podStartE2EDuration="7.864922841s" totalImagesPullingTime="5.849075685s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:40.864234623 +0000 UTC m=+331.216669689" watchObservedRunningTime="2026-07-09 18:58:40.864922841 +0000 UTC m=+331.217357909"
Jul 09 18:58:40.912181 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.912121    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf" podStartSLOduration=2.463991416 podStartE2EDuration="6.912101474s" totalImagesPullingTime="4.448110058s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:40.911874137 +0000 UTC m=+331.264309204" watchObservedRunningTime="2026-07-09 18:58:40.912101474 +0000 UTC m=+331.264536543"
Jul 09 18:58:40.913545 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:40.913488    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6" podStartSLOduration=3.044950631 podStartE2EDuration="6.913474503s" totalImagesPullingTime="3.868523872s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:40.886485102 +0000 UTC m=+331.238920167" watchObservedRunningTime="2026-07-09 18:58:40.913474503 +0000 UTC m=+331.265909571"
Jul 09 18:58:41.050548 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:41.050466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:41.155428 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:41.155331    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:41.462352 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:41.462315    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:41.570370 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:41.570337    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:41.830299 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:41.830224    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:41.970864 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:41.970830    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:44.114560 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.114511    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:44.275493 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.275458    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"]
Jul 09 18:58:44.279092 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.279074    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.282683 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.282662    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-79djv\""
Jul 09 18:58:44.283810 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.283792    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 09 18:58:44.335085 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.335054    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:44.357813 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.357780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5rr6d\" (UniqueName: \"kubernetes.io/projected/d6d2f532-9346-4908-a5d8-3a06cea409b0-kube-api-access-5rr6d\") pod \"kserve-controller-manager-d6f4b65b8-57w4c\" (UID: \"d6d2f532-9346-4908-a5d8-3a06cea409b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.357978 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.357843    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d2f532-9346-4908-a5d8-3a06cea409b0-cert\") pod \"kserve-controller-manager-d6f4b65b8-57w4c\" (UID: \"d6d2f532-9346-4908-a5d8-3a06cea409b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.364797 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.364724    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"]
Jul 09 18:58:44.425728 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.425690    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:44.458907 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.458876    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d2f532-9346-4908-a5d8-3a06cea409b0-cert\") pod \"kserve-controller-manager-d6f4b65b8-57w4c\" (UID: \"d6d2f532-9346-4908-a5d8-3a06cea409b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.459060 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.458939    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5rr6d\" (UniqueName: \"kubernetes.io/projected/d6d2f532-9346-4908-a5d8-3a06cea409b0-kube-api-access-5rr6d\") pod \"kserve-controller-manager-d6f4b65b8-57w4c\" (UID: \"d6d2f532-9346-4908-a5d8-3a06cea409b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.461395 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.461372    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/d6d2f532-9346-4908-a5d8-3a06cea409b0-cert\") pod \"kserve-controller-manager-d6f4b65b8-57w4c\" (UID: \"d6d2f532-9346-4908-a5d8-3a06cea409b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.473658 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.473629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5rr6d\" (UniqueName: \"kubernetes.io/projected/d6d2f532-9346-4908-a5d8-3a06cea409b0-kube-api-access-5rr6d\") pod \"kserve-controller-manager-d6f4b65b8-57w4c\" (UID: \"d6d2f532-9346-4908-a5d8-3a06cea409b0\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.525907 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.525876    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:44.598444 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.589861    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:44.754867 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.754845    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"]
Jul 09 18:58:44.756581 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:58:44.756554    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6d2f532_9346_4908_a5d8_3a06cea409b0.slice/crio-abba0234d02a6fb4a64d86ac60e93aabfed05a3d5229ad5f26ef472d74750497 WatchSource:0}: Error finding container abba0234d02a6fb4a64d86ac60e93aabfed05a3d5229ad5f26ef472d74750497: Status 404 returned error can't find the container with id abba0234d02a6fb4a64d86ac60e93aabfed05a3d5229ad5f26ef472d74750497
Jul 09 18:58:44.843287 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:44.843257    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c" event={"ID":"d6d2f532-9346-4908-a5d8-3a06cea409b0","Type":"ContainerStarted","Data":"abba0234d02a6fb4a64d86ac60e93aabfed05a3d5229ad5f26ef472d74750497"}
Jul 09 18:58:45.606128 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:45.606096    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:45.784881 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:45.784843    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:45.970946 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:45.970913    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:58:47.859611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:47.859580    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c" event={"ID":"d6d2f532-9346-4908-a5d8-3a06cea409b0","Type":"ContainerStarted","Data":"9803a71e7406a77f38fb7f3116a12daca3b67883c8ede7ae4c66c8fd5e18b9a9"}
Jul 09 18:58:47.860011 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:47.859667    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:58:47.880321 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:47.880275    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c" podStartSLOduration=1.345267266 podStartE2EDuration="3.88026094s" totalImagesPullingTime="2.534993674s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:58:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:58:47.878558241 +0000 UTC m=+338.230993302" watchObservedRunningTime="2026-07-09 18:58:47.88026094 +0000 UTC m=+338.232696004"
Jul 09 18:58:49.408254 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.408226    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"]
Jul 09 18:58:49.408636 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.408432    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf" podUID="44ffbb3e-ef70-47a4-bcbd-21199d473045" containerName="discovery" containerID="cri-o://06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931" gracePeriod=30
Jul 09 18:58:49.659503 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.659443    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:49.805766 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.805734    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ffbb3e-ef70-47a4-bcbd-21199d473045-local-certs\") pod \"44ffbb3e-ef70-47a4-bcbd-21199d473045\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") "
Jul 09 18:58:49.805955 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.805782    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-kubeconfig\") pod \"44ffbb3e-ef70-47a4-bcbd-21199d473045\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") "
Jul 09 18:58:49.805955 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.805821    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4tsh2\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-kube-api-access-4tsh2\") pod \"44ffbb3e-ef70-47a4-bcbd-21199d473045\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") "
Jul 09 18:58:49.805955 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.805843    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-cacerts\") pod \"44ffbb3e-ef70-47a4-bcbd-21199d473045\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") "
Jul 09 18:58:49.805955 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.805874    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-token\") pod \"44ffbb3e-ef70-47a4-bcbd-21199d473045\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") "
Jul 09 18:58:49.805955 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.805922    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-ca-configmap\") pod \"44ffbb3e-ef70-47a4-bcbd-21199d473045\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") "
Jul 09 18:58:49.805955 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.805953    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-dns-cert\") pod \"44ffbb3e-ef70-47a4-bcbd-21199d473045\" (UID: \"44ffbb3e-ef70-47a4-bcbd-21199d473045\") "
Jul 09 18:58:49.806397 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.806350    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "44ffbb3e-ef70-47a4-bcbd-21199d473045" (UID: "44ffbb3e-ef70-47a4-bcbd-21199d473045"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 18:58:49.808285 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.808250    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "44ffbb3e-ef70-47a4-bcbd-21199d473045" (UID: "44ffbb3e-ef70-47a4-bcbd-21199d473045"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:58:49.808422 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.808317    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-kube-api-access-4tsh2" (OuterVolumeSpecName: "kube-api-access-4tsh2") pod "44ffbb3e-ef70-47a4-bcbd-21199d473045" (UID: "44ffbb3e-ef70-47a4-bcbd-21199d473045"). InnerVolumeSpecName "kube-api-access-4tsh2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:58:49.808422 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.808317    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/44ffbb3e-ef70-47a4-bcbd-21199d473045-local-certs" (OuterVolumeSpecName: "local-certs") pod "44ffbb3e-ef70-47a4-bcbd-21199d473045" (UID: "44ffbb3e-ef70-47a4-bcbd-21199d473045"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 18:58:49.808422 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.808354    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-cacerts" (OuterVolumeSpecName: "cacerts") pod "44ffbb3e-ef70-47a4-bcbd-21199d473045" (UID: "44ffbb3e-ef70-47a4-bcbd-21199d473045"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:58:49.808614 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.808568    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "44ffbb3e-ef70-47a4-bcbd-21199d473045" (UID: "44ffbb3e-ef70-47a4-bcbd-21199d473045"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 18:58:49.808669 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.808633    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-token" (OuterVolumeSpecName: "istio-token") pod "44ffbb3e-ef70-47a4-bcbd-21199d473045" (UID: "44ffbb3e-ef70-47a4-bcbd-21199d473045"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:58:49.870041 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.870010    2556 generic.go:356] "Generic (PLEG): container finished" podID="44ffbb3e-ef70-47a4-bcbd-21199d473045" containerID="06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931" exitCode=0
Jul 09 18:58:49.870195 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.870057    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"
Jul 09 18:58:49.870195 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.870058    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf" event={"ID":"44ffbb3e-ef70-47a4-bcbd-21199d473045","Type":"ContainerDied","Data":"06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931"}
Jul 09 18:58:49.870195 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.870181    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf" event={"ID":"44ffbb3e-ef70-47a4-bcbd-21199d473045","Type":"ContainerDied","Data":"6611bd52c71733e06b29d13b7009e0c22f39f4bb7fdaa9c320fcf99616705115"}
Jul 09 18:58:49.870289 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.870212    2556 scope.go:117] "RemoveContainer" containerID="06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931"
Jul 09 18:58:49.879406 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.879388    2556 scope.go:117] "RemoveContainer" containerID="06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931"
Jul 09 18:58:49.879692 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:58:49.879674    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931\": container with ID starting with 06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931 not found: ID does not exist" containerID="06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931"
Jul 09 18:58:49.879746 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.879702    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931"} err="failed to get container status \"06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931\": rpc error: code = NotFound desc = could not find container \"06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931\": container with ID starting with 06e5307b79329805e5b5b3b1591da94ce2a0f78d1da0934cf04e7c9d97bf8931 not found: ID does not exist"
Jul 09 18:58:49.896222 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.896193    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"]
Jul 09 18:58:49.898989 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.898967    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7574459656-nj8mf"]
Jul 09 18:58:49.906553 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.906516    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-token\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.906611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.906557    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-ca-configmap\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.906611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.906566    2556 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-csr-dns-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.906611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.906575    2556 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/44ffbb3e-ef70-47a4-bcbd-21199d473045-local-certs\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.906611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.906585    2556 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-istio-kubeconfig\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.906611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.906594    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4tsh2\" (UniqueName: \"kubernetes.io/projected/44ffbb3e-ef70-47a4-bcbd-21199d473045-kube-api-access-4tsh2\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:58:49.906611 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:49.906601    2556 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/44ffbb3e-ef70-47a4-bcbd-21199d473045-cacerts\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:58:50.084044 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:50.084012    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44ffbb3e-ef70-47a4-bcbd-21199d473045" path="/var/lib/kubelet/pods/44ffbb3e-ef70-47a4-bcbd-21199d473045/volumes"
Jul 09 18:58:51.831700 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:51.831674    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-j97g2"
Jul 09 18:58:51.833613 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:58:51.833592    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-p8rb6"
Jul 09 18:59:15.602865 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:15.602829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:59:15.668261 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:15.668227    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:59:18.869445 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:18.869407    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-57w4c"
Jul 09 18:59:34.082675 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.082643    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:59:34.196751 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.196716    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:59:34.222920 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.222885    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4p5kz"]
Jul 09 18:59:34.223395 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.223375    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44ffbb3e-ef70-47a4-bcbd-21199d473045" containerName="discovery"
Jul 09 18:59:34.223395 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.223394    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="44ffbb3e-ef70-47a4-bcbd-21199d473045" containerName="discovery"
Jul 09 18:59:34.223600 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.223467    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="44ffbb3e-ef70-47a4-bcbd-21199d473045" containerName="discovery"
Jul 09 18:59:34.226478 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.226461    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz"
Jul 09 18:59:34.229134 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.229114    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 09 18:59:34.229541 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.229169    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 09 18:59:34.229978 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.229963    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-vn7v8\""
Jul 09 18:59:34.240059 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.240033    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4p5kz"]
Jul 09 18:59:34.280853 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.280820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:59:34.378908 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.378822    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqxjk\" (UniqueName: \"kubernetes.io/projected/45d98f62-9c6a-4d52-92cb-a90458fe809c-kube-api-access-fqxjk\") pod \"kuadrant-operator-catalog-4p5kz\" (UID: \"45d98f62-9c6a-4d52-92cb-a90458fe809c\") " pod="kuadrant-system/kuadrant-operator-catalog-4p5kz"
Jul 09 18:59:34.479906 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.479862    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqxjk\" (UniqueName: \"kubernetes.io/projected/45d98f62-9c6a-4d52-92cb-a90458fe809c-kube-api-access-fqxjk\") pod \"kuadrant-operator-catalog-4p5kz\" (UID: \"45d98f62-9c6a-4d52-92cb-a90458fe809c\") " pod="kuadrant-system/kuadrant-operator-catalog-4p5kz"
Jul 09 18:59:34.488856 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.488831    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqxjk\" (UniqueName: \"kubernetes.io/projected/45d98f62-9c6a-4d52-92cb-a90458fe809c-kube-api-access-fqxjk\") pod \"kuadrant-operator-catalog-4p5kz\" (UID: \"45d98f62-9c6a-4d52-92cb-a90458fe809c\") " pod="kuadrant-system/kuadrant-operator-catalog-4p5kz"
Jul 09 18:59:34.538934 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.538896    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz"
Jul 09 18:59:34.547902 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.547876    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4p5kz"]
Jul 09 18:59:34.675448 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.675423    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4p5kz"]
Jul 09 18:59:34.677569 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:59:34.677540    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45d98f62_9c6a_4d52_92cb_a90458fe809c.slice/crio-9a135614e5ebaaae24aecb203130024178000806fc9cbf38b75268d3823575b9 WatchSource:0}: Error finding container 9a135614e5ebaaae24aecb203130024178000806fc9cbf38b75268d3823575b9: Status 404 returned error can't find the container with id 9a135614e5ebaaae24aecb203130024178000806fc9cbf38b75268d3823575b9
Jul 09 18:59:34.761855 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.761823    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fhjm2"]
Jul 09 18:59:34.765972 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.765952    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:34.776081 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.776057    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fhjm2"]
Jul 09 18:59:34.883283 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.883243    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sld6b\" (UniqueName: \"kubernetes.io/projected/fb30b66d-1c28-47e0-aeb8-257d0852e646-kube-api-access-sld6b\") pod \"kuadrant-operator-catalog-fhjm2\" (UID: \"fb30b66d-1c28-47e0-aeb8-257d0852e646\") " pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:34.984459 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.984374    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sld6b\" (UniqueName: \"kubernetes.io/projected/fb30b66d-1c28-47e0-aeb8-257d0852e646-kube-api-access-sld6b\") pod \"kuadrant-operator-catalog-fhjm2\" (UID: \"fb30b66d-1c28-47e0-aeb8-257d0852e646\") " pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:34.986131 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.986089    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:59:34.994456 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:34.994429    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sld6b\" (UniqueName: \"kubernetes.io/projected/fb30b66d-1c28-47e0-aeb8-257d0852e646-kube-api-access-sld6b\") pod \"kuadrant-operator-catalog-fhjm2\" (UID: \"fb30b66d-1c28-47e0-aeb8-257d0852e646\") " pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:35.075891 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:35.075827    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:35.081206 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:35.081173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz" event={"ID":"45d98f62-9c6a-4d52-92cb-a90458fe809c","Type":"ContainerStarted","Data":"9a135614e5ebaaae24aecb203130024178000806fc9cbf38b75268d3823575b9"}
Jul 09 18:59:35.198801 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:35.198768    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 18:59:35.216114 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:35.216090    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fhjm2"]
Jul 09 18:59:35.217966 ip-10-0-129-93 kubenswrapper[2556]: W0709 18:59:35.217935    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb30b66d_1c28_47e0_aeb8_257d0852e646.slice/crio-f6eb0dfcc316e0f5331b80bc6a812d96f21434ef072d79c7a391204de586b3fb WatchSource:0}: Error finding container f6eb0dfcc316e0f5331b80bc6a812d96f21434ef072d79c7a391204de586b3fb: Status 404 returned error can't find the container with id f6eb0dfcc316e0f5331b80bc6a812d96f21434ef072d79c7a391204de586b3fb
Jul 09 18:59:36.087890 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:36.087839    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2" event={"ID":"fb30b66d-1c28-47e0-aeb8-257d0852e646","Type":"ContainerStarted","Data":"f6eb0dfcc316e0f5331b80bc6a812d96f21434ef072d79c7a391204de586b3fb"}
Jul 09 18:59:37.094164 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.094127    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2" event={"ID":"fb30b66d-1c28-47e0-aeb8-257d0852e646","Type":"ContainerStarted","Data":"2deaa14cd0fa53d7d66d80ea6934b613aec25b5b5a6876b7dbfc9cbc9a848544"}
Jul 09 18:59:37.095734 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.095712    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz" event={"ID":"45d98f62-9c6a-4d52-92cb-a90458fe809c","Type":"ContainerStarted","Data":"81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b"}
Jul 09 18:59:37.095844 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.095800    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz" podUID="45d98f62-9c6a-4d52-92cb-a90458fe809c" containerName="registry-server" containerID="cri-o://81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b" gracePeriod=2
Jul 09 18:59:37.113129 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.113085    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2" podStartSLOduration=1.428795336 podStartE2EDuration="3.11306992s" totalImagesPullingTime="1.684274584s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:37.110518986 +0000 UTC m=+387.462954050" watchObservedRunningTime="2026-07-09 18:59:37.11306992 +0000 UTC m=+387.465504987"
Jul 09 18:59:37.128695 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.128640    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz" podStartSLOduration=0.907001507 podStartE2EDuration="3.128625896s" totalImagesPullingTime="2.221624389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 18:59:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 18:59:37.128105084 +0000 UTC m=+387.480540148" watchObservedRunningTime="2026-07-09 18:59:37.128625896 +0000 UTC m=+387.481060961"
Jul 09 18:59:37.351637 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.351611    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz"
Jul 09 18:59:37.508981 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.508883    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fqxjk\" (UniqueName: \"kubernetes.io/projected/45d98f62-9c6a-4d52-92cb-a90458fe809c-kube-api-access-fqxjk\") pod \"45d98f62-9c6a-4d52-92cb-a90458fe809c\" (UID: \"45d98f62-9c6a-4d52-92cb-a90458fe809c\") "
Jul 09 18:59:37.511062 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.511029    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45d98f62-9c6a-4d52-92cb-a90458fe809c-kube-api-access-fqxjk" (OuterVolumeSpecName: "kube-api-access-fqxjk") pod "45d98f62-9c6a-4d52-92cb-a90458fe809c" (UID: "45d98f62-9c6a-4d52-92cb-a90458fe809c"). InnerVolumeSpecName "kube-api-access-fqxjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 18:59:37.609436 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:37.609399    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqxjk\" (UniqueName: \"kubernetes.io/projected/45d98f62-9c6a-4d52-92cb-a90458fe809c-kube-api-access-fqxjk\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 18:59:38.101321 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.101290    2556 generic.go:356] "Generic (PLEG): container finished" podID="45d98f62-9c6a-4d52-92cb-a90458fe809c" containerID="81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b" exitCode=0
Jul 09 18:59:38.101687 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.101334    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz"
Jul 09 18:59:38.101687 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.101367    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz" event={"ID":"45d98f62-9c6a-4d52-92cb-a90458fe809c","Type":"ContainerDied","Data":"81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b"}
Jul 09 18:59:38.101687 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.101416    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4p5kz" event={"ID":"45d98f62-9c6a-4d52-92cb-a90458fe809c","Type":"ContainerDied","Data":"9a135614e5ebaaae24aecb203130024178000806fc9cbf38b75268d3823575b9"}
Jul 09 18:59:38.101687 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.101436    2556 scope.go:117] "RemoveContainer" containerID="81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b"
Jul 09 18:59:38.110846 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.110829    2556 scope.go:117] "RemoveContainer" containerID="81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b"
Jul 09 18:59:38.111107 ip-10-0-129-93 kubenswrapper[2556]: E0709 18:59:38.111089    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b\": container with ID starting with 81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b not found: ID does not exist" containerID="81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b"
Jul 09 18:59:38.111167 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.111115    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b"} err="failed to get container status \"81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b\": rpc error: code = NotFound desc = could not find container \"81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b\": container with ID starting with 81f71322581ed2908374b7fb600ddc654ff4dd53da52413b1ef91cf144303c1b not found: ID does not exist"
Jul 09 18:59:38.118769 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.118744    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4p5kz"]
Jul 09 18:59:38.121156 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:38.121133    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4p5kz"]
Jul 09 18:59:40.084342 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:40.084305    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45d98f62-9c6a-4d52-92cb-a90458fe809c" path="/var/lib/kubelet/pods/45d98f62-9c6a-4d52-92cb-a90458fe809c/volumes"
Jul 09 18:59:45.076650 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:45.076616    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:45.077241 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:45.076674    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:45.099205 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:45.099180    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 18:59:45.155693 ip-10-0-129-93 kubenswrapper[2556]: I0709 18:59:45.155656    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-fhjm2"
Jul 09 19:00:00.056771 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.056742    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-6fb5bb4d87-htgxd"]
Jul 09 19:00:00.057151 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.057137    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45d98f62-9c6a-4d52-92cb-a90458fe809c" containerName="registry-server"
Jul 09 19:00:00.057207 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.057154    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="45d98f62-9c6a-4d52-92cb-a90458fe809c" containerName="registry-server"
Jul 09 19:00:00.057258 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.057230    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="45d98f62-9c6a-4d52-92cb-a90458fe809c" containerName="registry-server"
Jul 09 19:00:00.061806 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.061786    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.065644 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.065619    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-cplfs\""
Jul 09 19:00:00.067220 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.067197    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Jul 09 19:00:00.067350 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.067336    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Jul 09 19:00:00.068372 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.068351    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Jul 09 19:00:00.068501 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.068351    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Jul 09 19:00:00.068618 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.068604    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Jul 09 19:00:00.075120 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.075100    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\""
Jul 09 19:00:00.083148 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.083122    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-serving-cert\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.083290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.083156    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-service-ca\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.083290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.083189    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7jvm\" (UniqueName: \"kubernetes.io/projected/18944fe4-9b38-4ca1-a9bf-be2e301da711-kube-api-access-b7jvm\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.083290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.083241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-oauth-config\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.083290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.083266    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-oauth-serving-cert\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.083484 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.083327    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-trusted-ca-bundle\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.083484 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.083360    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-config\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.084915 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.084894    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb5bb4d87-htgxd"]
Jul 09 19:00:00.127072 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.127040    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:00.183760 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.183724    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-oauth-config\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.183760 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.183763    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-oauth-serving-cert\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184008 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.183807    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-trusted-ca-bundle\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184008 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.183827    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-config\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184008 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.183880    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-serving-cert\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184008 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.183897    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-service-ca\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184008 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.183980    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7jvm\" (UniqueName: \"kubernetes.io/projected/18944fe4-9b38-4ca1-a9bf-be2e301da711-kube-api-access-b7jvm\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184772 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.184749    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-oauth-serving-cert\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184891 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.184778    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-service-ca\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184891 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.184792    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-config\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.184891 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.184816    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18944fe4-9b38-4ca1-a9bf-be2e301da711-trusted-ca-bundle\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.186775 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.186754    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-oauth-config\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.186981 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.186960    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/18944fe4-9b38-4ca1-a9bf-be2e301da711-console-serving-cert\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.192431 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.192408    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7jvm\" (UniqueName: \"kubernetes.io/projected/18944fe4-9b38-4ca1-a9bf-be2e301da711-kube-api-access-b7jvm\") pod \"console-6fb5bb4d87-htgxd\" (UID: \"18944fe4-9b38-4ca1-a9bf-be2e301da711\") " pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.218332 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.218295    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:00.372178 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.372087    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:00.514854 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.514817    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6fb5bb4d87-htgxd"]
Jul 09 19:00:00.518239 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:00:00.518208    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18944fe4_9b38_4ca1_a9bf_be2e301da711.slice/crio-9a55df067059ba715f038906d1a24a18625e60159c325ab5bac53b64618cd168 WatchSource:0}: Error finding container 9a55df067059ba715f038906d1a24a18625e60159c325ab5bac53b64618cd168: Status 404 returned error can't find the container with id 9a55df067059ba715f038906d1a24a18625e60159c325ab5bac53b64618cd168
Jul 09 19:00:00.984804 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:00.984767    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:01.206808 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:01.206774    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb5bb4d87-htgxd" event={"ID":"18944fe4-9b38-4ca1-a9bf-be2e301da711","Type":"ContainerStarted","Data":"728f20392fc29fc9892e4243284b293dfedb4935db610000803f54931b735d38"}
Jul 09 19:00:01.207263 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:01.206831    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6fb5bb4d87-htgxd" event={"ID":"18944fe4-9b38-4ca1-a9bf-be2e301da711","Type":"ContainerStarted","Data":"9a55df067059ba715f038906d1a24a18625e60159c325ab5bac53b64618cd168"}
Jul 09 19:00:01.228578 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:01.228463    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-6fb5bb4d87-htgxd" podStartSLOduration=1.228443156 podStartE2EDuration="1.228443156s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:01.227720582 +0000 UTC m=+411.580155647" watchObservedRunningTime="2026-07-09 19:00:01.228443156 +0000 UTC m=+411.580878222"
Jul 09 19:00:02.655986 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:02.655953    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:02.720043 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:02.719997    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:03.216415 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.216386    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"]
Jul 09 19:00:03.218504 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.218488    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:03.221763 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.221745    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-dmjfw\""
Jul 09 19:00:03.231664 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.231631    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"]
Jul 09 19:00:03.307953 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.307893    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/079314af-1d6b-4fd5-a71a-838914ab6060-kube-api-access-wnr2s\") pod \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" (UID: \"079314af-1d6b-4fd5-a71a-838914ab6060\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:03.408432 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.408391    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/079314af-1d6b-4fd5-a71a-838914ab6060-kube-api-access-wnr2s\") pod \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" (UID: \"079314af-1d6b-4fd5-a71a-838914ab6060\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:03.421223 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.421193    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/079314af-1d6b-4fd5-a71a-838914ab6060-kube-api-access-wnr2s\") pod \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" (UID: \"079314af-1d6b-4fd5-a71a-838914ab6060\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:03.529370 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.529274    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:03.667548 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:03.667482    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"]
Jul 09 19:00:03.670349 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:00:03.670326    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod079314af_1d6b_4fd5_a71a_838914ab6060.slice/crio-4d38e20f5ead6c21d8c73b6f7c02ee554a735cb60eff12143713181756dbe441 WatchSource:0}: Error finding container 4d38e20f5ead6c21d8c73b6f7c02ee554a735cb60eff12143713181756dbe441: Status 404 returned error can't find the container with id 4d38e20f5ead6c21d8c73b6f7c02ee554a735cb60eff12143713181756dbe441
Jul 09 19:00:04.221168 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:04.221132    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds" event={"ID":"079314af-1d6b-4fd5-a71a-838914ab6060","Type":"ContainerStarted","Data":"4d38e20f5ead6c21d8c73b6f7c02ee554a735cb60eff12143713181756dbe441"}
Jul 09 19:00:04.777428 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:04.777391    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:06.059970 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:06.059934    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:07.002084 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.002054    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"]
Jul 09 19:00:07.005664 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.005648    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"
Jul 09 19:00:07.009651 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.009632    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-brw67\""
Jul 09 19:00:07.010091 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.010077    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 09 19:00:07.037662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.037634    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"]
Jul 09 19:00:07.039588 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.039564    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ln7h8\" (UniqueName: \"kubernetes.io/projected/4d3243d2-2bf7-431c-a3a1-5f40a1325f08-kube-api-access-ln7h8\") pod \"dns-operator-controller-manager-864dd78c4c-m4shg\" (UID: \"4d3243d2-2bf7-431c-a3a1-5f40a1325f08\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"
Jul 09 19:00:07.140890 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.140851    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ln7h8\" (UniqueName: \"kubernetes.io/projected/4d3243d2-2bf7-431c-a3a1-5f40a1325f08-kube-api-access-ln7h8\") pod \"dns-operator-controller-manager-864dd78c4c-m4shg\" (UID: \"4d3243d2-2bf7-431c-a3a1-5f40a1325f08\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"
Jul 09 19:00:07.153923 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.153893    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ln7h8\" (UniqueName: \"kubernetes.io/projected/4d3243d2-2bf7-431c-a3a1-5f40a1325f08-kube-api-access-ln7h8\") pod \"dns-operator-controller-manager-864dd78c4c-m4shg\" (UID: \"4d3243d2-2bf7-431c-a3a1-5f40a1325f08\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"
Jul 09 19:00:07.238282 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.238247    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds" event={"ID":"079314af-1d6b-4fd5-a71a-838914ab6060","Type":"ContainerStarted","Data":"19878a6d6d3b861d4ec5dad19ae7fd6cb5702431e5d30dd1f31d630fe6c1bc75"}
Jul 09 19:00:07.238453 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.238308    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:07.286830 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.286711    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds" podStartSLOduration=1.597933864 podStartE2EDuration="4.286694018s" totalImagesPullingTime="2.688760154s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:07.284777303 +0000 UTC m=+417.637212369" watchObservedRunningTime="2026-07-09 19:00:07.286694018 +0000 UTC m=+417.639129083"
Jul 09 19:00:07.315760 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.315721    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"
Jul 09 19:00:07.453183 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.453160    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"]
Jul 09 19:00:07.455267 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:00:07.455240    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d3243d2_2bf7_431c_a3a1_5f40a1325f08.slice/crio-80f7cd4858e0835f919508b6a01bdbe0858ae9eeab42442f1fdf1646e7040c54 WatchSource:0}: Error finding container 80f7cd4858e0835f919508b6a01bdbe0858ae9eeab42442f1fdf1646e7040c54: Status 404 returned error can't find the container with id 80f7cd4858e0835f919508b6a01bdbe0858ae9eeab42442f1fdf1646e7040c54
Jul 09 19:00:07.798294 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:07.798263    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:08.244118 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:08.244086    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg" event={"ID":"4d3243d2-2bf7-431c-a3a1-5f40a1325f08","Type":"ContainerStarted","Data":"80f7cd4858e0835f919508b6a01bdbe0858ae9eeab42442f1fdf1646e7040c54"}
Jul 09 19:00:09.167103 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:09.167073    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:10.255463 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:10.255423    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg" event={"ID":"4d3243d2-2bf7-431c-a3a1-5f40a1325f08","Type":"ContainerStarted","Data":"cb448dba442922c29b8803320d04e93a38b4672f03c5cfa54c76d1a60023a0e3"}
Jul 09 19:00:10.255924 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:10.255564    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"
Jul 09 19:00:10.340845 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:10.340793    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg" podStartSLOduration=1.894022096 podStartE2EDuration="4.340776285s" totalImagesPullingTime="2.446754189s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:10.32522175 +0000 UTC m=+420.677656814" watchObservedRunningTime="2026-07-09 19:00:10.340776285 +0000 UTC m=+420.693211349"
Jul 09 19:00:10.372771 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:10.372742    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:10.372974 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:10.372894    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:10.377827 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:10.377804    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:10.391805 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:10.391772    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:11.264349 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:11.264320    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6fb5bb4d87-htgxd"
Jul 09 19:00:11.956736 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:11.956696    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:12.054833 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.054793    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:12.180851 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.180819    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"]
Jul 09 19:00:12.185799 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.185774    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.190654 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.190635    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zq2wt\""
Jul 09 19:00:12.201357 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.201334    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"]
Jul 09 19:00:12.287267 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.287175    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gnw4j\" (UniqueName: \"kubernetes.io/projected/07bc48ff-face-43b3-83ca-a2100c242005-kube-api-access-gnw4j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.287267 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.287220    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bc48ff-face-43b3-83ca-a2100c242005-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.295694 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.295662    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:12.388098 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.388051    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bc48ff-face-43b3-83ca-a2100c242005-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.388287 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.388189    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gnw4j\" (UniqueName: \"kubernetes.io/projected/07bc48ff-face-43b3-83ca-a2100c242005-kube-api-access-gnw4j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.388462 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.388439    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bc48ff-face-43b3-83ca-a2100c242005-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.405789 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.405762    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gnw4j\" (UniqueName: \"kubernetes.io/projected/07bc48ff-face-43b3-83ca-a2100c242005-kube-api-access-gnw4j\") pod \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.448328 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.448291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:12.495870 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.495833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:12.657076 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.651700    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"]
Jul 09 19:00:12.658904 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:00:12.658873    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07bc48ff_face_43b3_83ca_a2100c242005.slice/crio-454df9ef06495a259cf56f20bf796b31ded3bcb08fb7b72f5428236fd7ee4735 WatchSource:0}: Error finding container 454df9ef06495a259cf56f20bf796b31ded3bcb08fb7b72f5428236fd7ee4735: Status 404 returned error can't find the container with id 454df9ef06495a259cf56f20bf796b31ded3bcb08fb7b72f5428236fd7ee4735
Jul 09 19:00:12.988369 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:12.988334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:13.271458 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:13.271372    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78" event={"ID":"07bc48ff-face-43b3-83ca-a2100c242005","Type":"ContainerStarted","Data":"454df9ef06495a259cf56f20bf796b31ded3bcb08fb7b72f5428236fd7ee4735"}
Jul 09 19:00:14.818682 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:14.818642    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:16.192459 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.192420    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"]
Jul 09 19:00:16.195853 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.195833    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"
Jul 09 19:00:16.199183 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.199165    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-mqstl\""
Jul 09 19:00:16.208115 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.208094    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"]
Jul 09 19:00:16.224365 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.224328    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6trn5\" (UniqueName: \"kubernetes.io/projected/34a6488d-1d4c-446e-a390-cbc98c38b25c-kube-api-access-6trn5\") pod \"authorino-operator-75cc8d5cb7-9w2mn\" (UID: \"34a6488d-1d4c-446e-a390-cbc98c38b25c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"
Jul 09 19:00:16.325721 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.325683    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6trn5\" (UniqueName: \"kubernetes.io/projected/34a6488d-1d4c-446e-a390-cbc98c38b25c-kube-api-access-6trn5\") pod \"authorino-operator-75cc8d5cb7-9w2mn\" (UID: \"34a6488d-1d4c-446e-a390-cbc98c38b25c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"
Jul 09 19:00:16.334580 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.334511    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6trn5\" (UniqueName: \"kubernetes.io/projected/34a6488d-1d4c-446e-a390-cbc98c38b25c-kube-api-access-6trn5\") pod \"authorino-operator-75cc8d5cb7-9w2mn\" (UID: \"34a6488d-1d4c-446e-a390-cbc98c38b25c\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"
Jul 09 19:00:16.509721 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.509628    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"
Jul 09 19:00:16.660945 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:16.660914    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"]
Jul 09 19:00:16.663539 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:00:16.663485    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34a6488d_1d4c_446e_a390_cbc98c38b25c.slice/crio-74332bbaeb50294e2f869ba33f67c78776be3d8c2df43e5324d9eb4027186573 WatchSource:0}: Error finding container 74332bbaeb50294e2f869ba33f67c78776be3d8c2df43e5324d9eb4027186573: Status 404 returned error can't find the container with id 74332bbaeb50294e2f869ba33f67c78776be3d8c2df43e5324d9eb4027186573
Jul 09 19:00:17.292505 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:17.292477    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn" event={"ID":"34a6488d-1d4c-446e-a390-cbc98c38b25c","Type":"ContainerStarted","Data":"74332bbaeb50294e2f869ba33f67c78776be3d8c2df43e5324d9eb4027186573"}
Jul 09 19:00:18.212923 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:18.212884    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:18.246398 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:18.246368    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:18.312811 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:18.312712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:19.188931 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:19.188901    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:19.289847 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:19.289814    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:19.306077 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:19.306045    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78" event={"ID":"07bc48ff-face-43b3-83ca-a2100c242005","Type":"ContainerStarted","Data":"60b4bc139c024002cd5ca40b22965cb5f3ed827e57d53cf1a1c494982737e598"}
Jul 09 19:00:19.306241 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:19.306153    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:19.337694 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:19.337639    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78" podStartSLOduration=0.876178359 podStartE2EDuration="7.337621715s" totalImagesPullingTime="6.461443356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:19.336566963 +0000 UTC m=+429.689002026" watchObservedRunningTime="2026-07-09 19:00:19.337621715 +0000 UTC m=+429.690056780"
Jul 09 19:00:21.263374 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:21.263340    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-m4shg"
Jul 09 19:00:21.318388 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:21.318355    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn" event={"ID":"34a6488d-1d4c-446e-a390-cbc98c38b25c","Type":"ContainerStarted","Data":"1a86a6411385fde0f5cfa32738be32d01abe91efad3a81a2dac5d7d7b4b2cc49"}
Jul 09 19:00:21.318588 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:21.318442    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"
Jul 09 19:00:21.341206 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:21.341154    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn" podStartSLOduration=1.379150262 podStartE2EDuration="5.34113735s" totalImagesPullingTime="3.961987088s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:21.341101125 +0000 UTC m=+431.693536190" watchObservedRunningTime="2026-07-09 19:00:21.34113735 +0000 UTC m=+431.693572414"
Jul 09 19:00:22.186263 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:22.186228    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:22.389643 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:22.389603    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:22.506406 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:22.506317    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:22.705950 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:22.705918    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:24.380956 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:24.380911    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:24.438139 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:24.438109    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:30.314514 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:30.314473    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.125396 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.125315    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"]
Jul 09 19:00:32.125807 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.125544    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78" podUID="07bc48ff-face-43b3-83ca-a2100c242005" containerName="manager" containerID="cri-o://60b4bc139c024002cd5ca40b22965cb5f3ed827e57d53cf1a1c494982737e598" gracePeriod=2
Jul 09 19:00:32.139207 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.139171    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"]
Jul 09 19:00:32.140111 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.140079    2556 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"07bc48ff-face-43b3-83ca-a2100c242005\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-l4s78\": pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.159277 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.159112    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"]
Jul 09 19:00:32.159700 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.159676    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="07bc48ff-face-43b3-83ca-a2100c242005" containerName="manager"
Jul 09 19:00:32.159700 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.159698    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="07bc48ff-face-43b3-83ca-a2100c242005" containerName="manager"
Jul 09 19:00:32.159865 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.159817    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="07bc48ff-face-43b3-83ca-a2100c242005" containerName="manager"
Jul 09 19:00:32.163223 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.163200    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"]
Jul 09 19:00:32.163339 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.163304    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.163419 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.163397    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" containerName="manager" containerID="cri-o://19878a6d6d3b861d4ec5dad19ae7fd6cb5702431e5d30dd1f31d630fe6c1bc75" gracePeriod=2
Jul 09 19:00:32.177673 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.177644    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"]
Jul 09 19:00:32.188413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.188381    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"]
Jul 09 19:00:32.189915 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.189882    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.200335 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.200311    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"]
Jul 09 19:00:32.200769 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.200751    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" containerName="manager"
Jul 09 19:00:32.200859 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.200770    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" containerName="manager"
Jul 09 19:00:32.200918 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.200902    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" containerName="manager"
Jul 09 19:00:32.204520 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.204499    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"
Jul 09 19:00:32.222357 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.222323    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"]
Jul 09 19:00:32.225161 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.225132    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:32.227486 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.227442    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.274469 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.274445    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj4w4\" (UniqueName: \"kubernetes.io/projected/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-kube-api-access-vj4w4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpdmf\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.274617 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.274596    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpdmf\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.274671 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.274659    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffc9b\" (UniqueName: \"kubernetes.io/projected/608b40b5-6d2b-41ea-a49e-489e44f78eab-kube-api-access-ffc9b\") pod \"limitador-operator-controller-manager-859d7b78bd-brm9r\" (UID: \"608b40b5-6d2b-41ea-a49e-489e44f78eab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"
Jul 09 19:00:32.326297 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.326267    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-9w2mn"
Jul 09 19:00:32.349401 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.349367    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:32.351672 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.351634    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.375325 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.375251    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpdmf\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.375325 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.375329    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffc9b\" (UniqueName: \"kubernetes.io/projected/608b40b5-6d2b-41ea-a49e-489e44f78eab-kube-api-access-ffc9b\") pod \"limitador-operator-controller-manager-859d7b78bd-brm9r\" (UID: \"608b40b5-6d2b-41ea-a49e-489e44f78eab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"
Jul 09 19:00:32.375745 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.375406    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj4w4\" (UniqueName: \"kubernetes.io/projected/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-kube-api-access-vj4w4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpdmf\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.375745 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.375688    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpdmf\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.377377 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.377353    2556 generic.go:356] "Generic (PLEG): container finished" podID="079314af-1d6b-4fd5-a71a-838914ab6060" containerID="19878a6d6d3b861d4ec5dad19ae7fd6cb5702431e5d30dd1f31d630fe6c1bc75" exitCode=0
Jul 09 19:00:32.379188 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.379158    2556 generic.go:356] "Generic (PLEG): container finished" podID="07bc48ff-face-43b3-83ca-a2100c242005" containerID="60b4bc139c024002cd5ca40b22965cb5f3ed827e57d53cf1a1c494982737e598" exitCode=0
Jul 09 19:00:32.384662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.384626    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj4w4\" (UniqueName: \"kubernetes.io/projected/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-kube-api-access-vj4w4\") pod \"kuadrant-operator-controller-manager-8684c9ff49-lpdmf\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.385194 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.385171    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffc9b\" (UniqueName: \"kubernetes.io/projected/608b40b5-6d2b-41ea-a49e-489e44f78eab-kube-api-access-ffc9b\") pod \"limitador-operator-controller-manager-859d7b78bd-brm9r\" (UID: \"608b40b5-6d2b-41ea-a49e-489e44f78eab\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"
Jul 09 19:00:32.419615 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.419589    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:32.422201 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.422170    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:32.424142 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.424121    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.424651 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.424626    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.426678 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.426656    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:32.428867 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:32.428847    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:32.565796 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.565762    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:32.572542 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.572516    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"
Jul 09 19:00:32.577461 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.577439    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gnw4j\" (UniqueName: \"kubernetes.io/projected/07bc48ff-face-43b3-83ca-a2100c242005-kube-api-access-gnw4j\") pod \"07bc48ff-face-43b3-83ca-a2100c242005\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") "
Jul 09 19:00:32.577559 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.577504    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/079314af-1d6b-4fd5-a71a-838914ab6060-kube-api-access-wnr2s\") pod \"079314af-1d6b-4fd5-a71a-838914ab6060\" (UID: \"079314af-1d6b-4fd5-a71a-838914ab6060\") "
Jul 09 19:00:32.577652 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.577637    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bc48ff-face-43b3-83ca-a2100c242005-extensions-socket-volume\") pod \"07bc48ff-face-43b3-83ca-a2100c242005\" (UID: \"07bc48ff-face-43b3-83ca-a2100c242005\") "
Jul 09 19:00:32.578270 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.578242    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07bc48ff-face-43b3-83ca-a2100c242005-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "07bc48ff-face-43b3-83ca-a2100c242005" (UID: "07bc48ff-face-43b3-83ca-a2100c242005"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:00:32.579624 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.579596    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/079314af-1d6b-4fd5-a71a-838914ab6060-kube-api-access-wnr2s" (OuterVolumeSpecName: "kube-api-access-wnr2s") pod "079314af-1d6b-4fd5-a71a-838914ab6060" (UID: "079314af-1d6b-4fd5-a71a-838914ab6060"). InnerVolumeSpecName "kube-api-access-wnr2s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:00:32.579903 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.579875    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07bc48ff-face-43b3-83ca-a2100c242005-kube-api-access-gnw4j" (OuterVolumeSpecName: "kube-api-access-gnw4j") pod "07bc48ff-face-43b3-83ca-a2100c242005" (UID: "07bc48ff-face-43b3-83ca-a2100c242005"). InnerVolumeSpecName "kube-api-access-gnw4j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:00:32.679195 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.679167    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/07bc48ff-face-43b3-83ca-a2100c242005-extensions-socket-volume\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:00:32.679195 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.679195    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gnw4j\" (UniqueName: \"kubernetes.io/projected/07bc48ff-face-43b3-83ca-a2100c242005-kube-api-access-gnw4j\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:00:32.679374 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.679208    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wnr2s\" (UniqueName: \"kubernetes.io/projected/079314af-1d6b-4fd5-a71a-838914ab6060-kube-api-access-wnr2s\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:00:32.743223 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.743188    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"]
Jul 09 19:00:32.746792 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:00:32.746759    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode09d8b2a_c0bd_4b3e_a8ce_1c33358d8341.slice/crio-08c2e8ef0f19033ae79db18a90e0c1b453d19b371b566b1958495f591ce6eb55 WatchSource:0}: Error finding container 08c2e8ef0f19033ae79db18a90e0c1b453d19b371b566b1958495f591ce6eb55: Status 404 returned error can't find the container with id 08c2e8ef0f19033ae79db18a90e0c1b453d19b371b566b1958495f591ce6eb55
Jul 09 19:00:32.765046 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:32.765024    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"]
Jul 09 19:00:32.766283 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:00:32.766252    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod608b40b5_6d2b_41ea_a49e_489e44f78eab.slice/crio-33df031051043a227ef075d7e749e5bf1ecc60c1d5fec5f98b89f39bfa324b0a WatchSource:0}: Error finding container 33df031051043a227ef075d7e749e5bf1ecc60c1d5fec5f98b89f39bfa324b0a: Status 404 returned error can't find the container with id 33df031051043a227ef075d7e749e5bf1ecc60c1d5fec5f98b89f39bfa324b0a
Jul 09 19:00:33.150089 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.150062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:33.235362 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.235326    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:33.335507 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.335466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:33.385520 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.385493    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:33.385520 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.385506    2556 scope.go:117] "RemoveContainer" containerID="19878a6d6d3b861d4ec5dad19ae7fd6cb5702431e5d30dd1f31d630fe6c1bc75"
Jul 09 19:00:33.387004 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.386986    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:33.388112 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:33.388081    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:33.388940 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.388915    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r" event={"ID":"608b40b5-6d2b-41ea-a49e-489e44f78eab","Type":"ContainerStarted","Data":"d575d88be471c8cd43df68b09a290b14d6eea45b5e53bae108ebcf52a53d3ba7"}
Jul 09 19:00:33.389054 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.388959    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r" event={"ID":"608b40b5-6d2b-41ea-a49e-489e44f78eab","Type":"ContainerStarted","Data":"33df031051043a227ef075d7e749e5bf1ecc60c1d5fec5f98b89f39bfa324b0a"}
Jul 09 19:00:33.389054 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.389002    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"
Jul 09 19:00:33.390451 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:33.390362    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:33.391125 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.391099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf" event={"ID":"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341","Type":"ContainerStarted","Data":"083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa"}
Jul 09 19:00:33.391232 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.391139    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf" event={"ID":"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341","Type":"ContainerStarted","Data":"08c2e8ef0f19033ae79db18a90e0c1b453d19b371b566b1958495f591ce6eb55"}
Jul 09 19:00:33.391355 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.391338    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:33.396195 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.396073    2556 scope.go:117] "RemoveContainer" containerID="60b4bc139c024002cd5ca40b22965cb5f3ed827e57d53cf1a1c494982737e598"
Jul 09 19:00:33.409874 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.409808    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r" podStartSLOduration=1.409794569 podStartE2EDuration="1.409794569s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:33.408845256 +0000 UTC m=+443.761280322" watchObservedRunningTime="2026-07-09 19:00:33.409794569 +0000 UTC m=+443.762229633"
Jul 09 19:00:33.410981 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:33.410956    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:33.413056 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:33.413036    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:33.437244 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:33.437194    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf" podStartSLOduration=1.437182637 podStartE2EDuration="1.437182637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:00:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:00:33.435261529 +0000 UTC m=+443.787696593" watchObservedRunningTime="2026-07-09 19:00:33.437182637 +0000 UTC m=+443.789617702"
Jul 09 19:00:33.438622 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:33.438595    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-cg4ds\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-cg4ds"
Jul 09 19:00:33.440486 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:00:33.440467    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-l4s78\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="07bc48ff-face-43b3-83ca-a2100c242005" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-l4s78"
Jul 09 19:00:34.085203 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:34.085158    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="079314af-1d6b-4fd5-a71a-838914ab6060" path="/var/lib/kubelet/pods/079314af-1d6b-4fd5-a71a-838914ab6060/volumes"
Jul 09 19:00:34.085647 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:34.085627    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="07bc48ff-face-43b3-83ca-a2100c242005" path="/var/lib/kubelet/pods/07bc48ff-face-43b3-83ca-a2100c242005/volumes"
Jul 09 19:00:35.854389 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:35.854351    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:36.083506 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:36.083471    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:38.208681 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:38.208640    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:38.310642 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:38.310609    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:42.410098 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:42.410062    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:42.487378 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:42.487328    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:44.400723 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:44.400693    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:00:44.401180 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:44.400763    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-brm9r"
Jul 09 19:00:52.403907 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:52.403873    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:52.467142 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:52.467112    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:53.001052 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:53.001011    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:00:53.104474 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:00:53.104441    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:03.728387 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:03.728353    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"]
Jul 09 19:01:03.728851 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:03.728589    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf" podUID="e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" containerName="manager" containerID="cri-o://083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa" gracePeriod=10
Jul 09 19:01:03.975347 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:03.975324    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:01:04.018205 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.018136    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"]
Jul 09 19:01:04.018537 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.018513    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" containerName="manager"
Jul 09 19:01:04.018585 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.018542    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" containerName="manager"
Jul 09 19:01:04.018632 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.018622    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" containerName="manager"
Jul 09 19:01:04.020408 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.020387    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.044552 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.044499    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"]
Jul 09 19:01:04.046647 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.046618    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-extensions-socket-volume\") pod \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") "
Jul 09 19:01:04.046905 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.046672    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj4w4\" (UniqueName: \"kubernetes.io/projected/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-kube-api-access-vj4w4\") pod \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\" (UID: \"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341\") "
Jul 09 19:01:04.046905 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.046878    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/156eb454-8dfa-4859-b6ba-0134e38cc87b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hwrxv\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.047121 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.047001    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/156eb454-8dfa-4859-b6ba-0134e38cc87b-kube-api-access-mds7f\") pod \"kuadrant-operator-controller-manager-86878c769f-hwrxv\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.047121 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.047039    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" (UID: "e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:01:04.048848 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.048819    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-kube-api-access-vj4w4" (OuterVolumeSpecName: "kube-api-access-vj4w4") pod "e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" (UID: "e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341"). InnerVolumeSpecName "kube-api-access-vj4w4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:01:04.147745 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.147704    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/156eb454-8dfa-4859-b6ba-0134e38cc87b-kube-api-access-mds7f\") pod \"kuadrant-operator-controller-manager-86878c769f-hwrxv\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.147941 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.147850    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/156eb454-8dfa-4859-b6ba-0134e38cc87b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hwrxv\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.147941 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.147929    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-extensions-socket-volume\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:01:04.148042 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.147942    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj4w4\" (UniqueName: \"kubernetes.io/projected/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341-kube-api-access-vj4w4\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:01:04.148255 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.148235    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/156eb454-8dfa-4859-b6ba-0134e38cc87b-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-hwrxv\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.156272 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.156245    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/156eb454-8dfa-4859-b6ba-0134e38cc87b-kube-api-access-mds7f\") pod \"kuadrant-operator-controller-manager-86878c769f-hwrxv\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.331403 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.331308    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:04.493729 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.493686    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"]
Jul 09 19:01:04.496872 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:01:04.496844    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod156eb454_8dfa_4859_b6ba_0134e38cc87b.slice/crio-55319d2d3b594551b85a274305f42d199e4694b79e65127bfe73c3f0223a07fc WatchSource:0}: Error finding container 55319d2d3b594551b85a274305f42d199e4694b79e65127bfe73c3f0223a07fc: Status 404 returned error can't find the container with id 55319d2d3b594551b85a274305f42d199e4694b79e65127bfe73c3f0223a07fc
Jul 09 19:01:04.554563 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.554513    2556 generic.go:356] "Generic (PLEG): container finished" podID="e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" containerID="083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa" exitCode=0
Jul 09 19:01:04.554683 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.554561    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf" event={"ID":"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341","Type":"ContainerDied","Data":"083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa"}
Jul 09 19:01:04.554683 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.554594    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"
Jul 09 19:01:04.554683 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.554640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf" event={"ID":"e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341","Type":"ContainerDied","Data":"08c2e8ef0f19033ae79db18a90e0c1b453d19b371b566b1958495f591ce6eb55"}
Jul 09 19:01:04.554683 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.554663    2556 scope.go:117] "RemoveContainer" containerID="083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa"
Jul 09 19:01:04.556598 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.556519    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv" event={"ID":"156eb454-8dfa-4859-b6ba-0134e38cc87b","Type":"ContainerStarted","Data":"55319d2d3b594551b85a274305f42d199e4694b79e65127bfe73c3f0223a07fc"}
Jul 09 19:01:04.563561 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.563540    2556 scope.go:117] "RemoveContainer" containerID="083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa"
Jul 09 19:01:04.563815 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:01:04.563796    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa\": container with ID starting with 083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa not found: ID does not exist" containerID="083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa"
Jul 09 19:01:04.563864 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.563824    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa"} err="failed to get container status \"083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa\": rpc error: code = NotFound desc = could not find container \"083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa\": container with ID starting with 083d2f217d4ff46e621a44cad738874517e973734dc1a4352eb522c77b7c67aa not found: ID does not exist"
Jul 09 19:01:04.580822 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.580787    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"]
Jul 09 19:01:04.586781 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:04.586705    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-lpdmf"]
Jul 09 19:01:05.563672 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:05.563640    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv" event={"ID":"156eb454-8dfa-4859-b6ba-0134e38cc87b","Type":"ContainerStarted","Data":"448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf"}
Jul 09 19:01:05.564106 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:05.563731    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:05.590870 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:05.590820    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv" podStartSLOduration=2.5908039670000003 podStartE2EDuration="2.590803967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:05.589892986 +0000 UTC m=+475.942328052" watchObservedRunningTime="2026-07-09 19:01:05.590803967 +0000 UTC m=+475.943239032"
Jul 09 19:01:06.085650 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:06.085616    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341" path="/var/lib/kubelet/pods/e09d8b2a-c0bd-4b3e-a8ce-1c33358d8341/volumes"
Jul 09 19:01:16.571653 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:16.571619    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:01:21.082420 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:21.082370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:21.183033 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:21.182985    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:22.150925 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.150892    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mf249"]
Jul 09 19:01:22.154564 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.154539    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.156957 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.156932    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 09 19:01:22.157072 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.156965    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-xcftl\""
Jul 09 19:01:22.163931 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.163895    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:22.164083 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.164004    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mf249"]
Jul 09 19:01:22.211956 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.211918    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/eb3946bf-465e-4255-983e-794bf199ad6b-kube-api-access-qn9c4\") pod \"limitador-limitador-64c94fd64f-mf249\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.211956 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.211957    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb3946bf-465e-4255-983e-794bf199ad6b-config-file\") pod \"limitador-limitador-64c94fd64f-mf249\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.259262 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.259225    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mf249"]
Jul 09 19:01:22.270921 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.270893    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:22.313168 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.313137    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/eb3946bf-465e-4255-983e-794bf199ad6b-kube-api-access-qn9c4\") pod \"limitador-limitador-64c94fd64f-mf249\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.313354 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.313176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb3946bf-465e-4255-983e-794bf199ad6b-config-file\") pod \"limitador-limitador-64c94fd64f-mf249\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.313899 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.313871    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb3946bf-465e-4255-983e-794bf199ad6b-config-file\") pod \"limitador-limitador-64c94fd64f-mf249\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.321518 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.321495    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/eb3946bf-465e-4255-983e-794bf199ad6b-kube-api-access-qn9c4\") pod \"limitador-limitador-64c94fd64f-mf249\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.383876 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.383840    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:22.466368 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.466337    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:22.610984 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.610958    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mf249"]
Jul 09 19:01:22.612945 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:01:22.612917    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3946bf_465e_4255_983e_794bf199ad6b.slice/crio-63e472419d7a08bcf6036245ca95f57b1cd4bebd801fece80911bbf80f2139a1 WatchSource:0}: Error finding container 63e472419d7a08bcf6036245ca95f57b1cd4bebd801fece80911bbf80f2139a1: Status 404 returned error can't find the container with id 63e472419d7a08bcf6036245ca95f57b1cd4bebd801fece80911bbf80f2139a1
Jul 09 19:01:22.654362 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.654332    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249" event={"ID":"eb3946bf-465e-4255-983e-794bf199ad6b","Type":"ContainerStarted","Data":"63e472419d7a08bcf6036245ca95f57b1cd4bebd801fece80911bbf80f2139a1"}
Jul 09 19:01:22.990175 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:22.990139    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:24.150406 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:24.150354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:25.675784 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:25.675743    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249" event={"ID":"eb3946bf-465e-4255-983e-794bf199ad6b","Type":"ContainerStarted","Data":"1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c"}
Jul 09 19:01:25.676219 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:25.675893    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:25.693894 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:25.693778    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249" podStartSLOduration=0.864904136 podStartE2EDuration="3.693764777s" totalImagesPullingTime="2.828860641s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:25.69261273 +0000 UTC m=+496.045047796" watchObservedRunningTime="2026-07-09 19:01:25.693764777 +0000 UTC m=+496.046199841"
Jul 09 19:01:26.768870 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:26.768829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:36.681741 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:36.681710    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:37.072691 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.072605    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mf249"]
Jul 09 19:01:37.072835 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.072783    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249" podUID="eb3946bf-465e-4255-983e-794bf199ad6b" containerName="limitador" containerID="cri-o://1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c" gracePeriod=30
Jul 09 19:01:37.621979 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.621950    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:37.736675 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.736634    2556 generic.go:356] "Generic (PLEG): container finished" podID="eb3946bf-465e-4255-983e-794bf199ad6b" containerID="1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c" exitCode=0
Jul 09 19:01:37.737110 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.736682    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249"
Jul 09 19:01:37.737110 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.736726    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249" event={"ID":"eb3946bf-465e-4255-983e-794bf199ad6b","Type":"ContainerDied","Data":"1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c"}
Jul 09 19:01:37.737110 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.736780    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-mf249" event={"ID":"eb3946bf-465e-4255-983e-794bf199ad6b","Type":"ContainerDied","Data":"63e472419d7a08bcf6036245ca95f57b1cd4bebd801fece80911bbf80f2139a1"}
Jul 09 19:01:37.737110 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.736802    2556 scope.go:117] "RemoveContainer" containerID="1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c"
Jul 09 19:01:37.744887 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.744870    2556 scope.go:117] "RemoveContainer" containerID="1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c"
Jul 09 19:01:37.745125 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:01:37.745107    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c\": container with ID starting with 1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c not found: ID does not exist" containerID="1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c"
Jul 09 19:01:37.745177 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.745135    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c"} err="failed to get container status \"1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c\": rpc error: code = NotFound desc = could not find container \"1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c\": container with ID starting with 1354ce5bd668f54dccb1bca6c48fae0e4c4f4555efd927a597516d84baf09a6c not found: ID does not exist"
Jul 09 19:01:37.747434 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.747416    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/eb3946bf-465e-4255-983e-794bf199ad6b-kube-api-access-qn9c4\") pod \"eb3946bf-465e-4255-983e-794bf199ad6b\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") "
Jul 09 19:01:37.747567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.747497    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb3946bf-465e-4255-983e-794bf199ad6b-config-file\") pod \"eb3946bf-465e-4255-983e-794bf199ad6b\" (UID: \"eb3946bf-465e-4255-983e-794bf199ad6b\") "
Jul 09 19:01:37.747804 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.747784    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb3946bf-465e-4255-983e-794bf199ad6b-config-file" (OuterVolumeSpecName: "config-file") pod "eb3946bf-465e-4255-983e-794bf199ad6b" (UID: "eb3946bf-465e-4255-983e-794bf199ad6b"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:01:37.749388 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.749371    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb3946bf-465e-4255-983e-794bf199ad6b-kube-api-access-qn9c4" (OuterVolumeSpecName: "kube-api-access-qn9c4") pod "eb3946bf-465e-4255-983e-794bf199ad6b" (UID: "eb3946bf-465e-4255-983e-794bf199ad6b"). InnerVolumeSpecName "kube-api-access-qn9c4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:01:37.848123 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.848088    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qn9c4\" (UniqueName: \"kubernetes.io/projected/eb3946bf-465e-4255-983e-794bf199ad6b-kube-api-access-qn9c4\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:01:37.848123 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:37.848115    2556 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/eb3946bf-465e-4255-983e-794bf199ad6b-config-file\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:01:38.060044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.060009    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mf249"]
Jul 09 19:01:38.065047 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.065023    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-mf249"]
Jul 09 19:01:38.084392 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.084362    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="eb3946bf-465e-4255-983e-794bf199ad6b" path="/var/lib/kubelet/pods/eb3946bf-465e-4255-983e-794bf199ad6b/volumes"
Jul 09 19:01:38.197725 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.197688    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:38.314114 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.313989    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:38.673994 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.673961    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"]
Jul 09 19:01:38.674433 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.674415    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="eb3946bf-465e-4255-983e-794bf199ad6b" containerName="limitador"
Jul 09 19:01:38.674556 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.674435    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="eb3946bf-465e-4255-983e-794bf199ad6b" containerName="limitador"
Jul 09 19:01:38.674556 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.674551    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="eb3946bf-465e-4255-983e-794bf199ad6b" containerName="limitador"
Jul 09 19:01:38.677771 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.677752    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:38.681344 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.681324    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 09 19:01:38.681429 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.681366    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 09 19:01:38.681429 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.681384    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 09 19:01:38.682311 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.682294    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-shszt\""
Jul 09 19:01:38.687128 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.687107    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"]
Jul 09 19:01:38.702890 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.702861    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:38.857409 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.857377    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kq7fs\" (UniqueName: \"kubernetes.io/projected/dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff-kube-api-access-kq7fs\") pod \"postgres-69fb4594cf-4s48q\" (UID: \"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:38.857801 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.857486    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff-data\") pod \"postgres-69fb4594cf-4s48q\" (UID: \"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:38.958661 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.958573    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff-data\") pod \"postgres-69fb4594cf-4s48q\" (UID: \"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:38.958661 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.958612    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kq7fs\" (UniqueName: \"kubernetes.io/projected/dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff-kube-api-access-kq7fs\") pod \"postgres-69fb4594cf-4s48q\" (UID: \"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:38.958969 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.958951    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff-data\") pod \"postgres-69fb4594cf-4s48q\" (UID: \"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:38.978516 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.978483    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kq7fs\" (UniqueName: \"kubernetes.io/projected/dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff-kube-api-access-kq7fs\") pod \"postgres-69fb4594cf-4s48q\" (UID: \"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:38.988324 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:38.988302    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:39.128569 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:39.128521    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"]
Jul 09 19:01:39.131809 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:01:39.131780    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddcbf8e6e_c9ab_4ca6_8b32_f631a9c1e8ff.slice/crio-08c1082b71ce06e3b4b1a9e900431fa813a196bc7f16f335b47ca2785b41c442 WatchSource:0}: Error finding container 08c1082b71ce06e3b4b1a9e900431fa813a196bc7f16f335b47ca2785b41c442: Status 404 returned error can't find the container with id 08c1082b71ce06e3b4b1a9e900431fa813a196bc7f16f335b47ca2785b41c442
Jul 09 19:01:39.749121 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:39.749087    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q" event={"ID":"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff","Type":"ContainerStarted","Data":"08c1082b71ce06e3b4b1a9e900431fa813a196bc7f16f335b47ca2785b41c442"}
Jul 09 19:01:45.786218 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:45.786180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q" event={"ID":"dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff","Type":"ContainerStarted","Data":"d03cee39c5141d21a7876d46da33cfe4f0464333843ed0f3b4fd660f3cec4250"}
Jul 09 19:01:45.786625 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:45.786256    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:45.807035 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:45.806986    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q" podStartSLOduration=1.286739943 podStartE2EDuration="7.80697098s" totalImagesPullingTime="6.520231037s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:01:45.805795778 +0000 UTC m=+516.158230845" watchObservedRunningTime="2026-07-09 19:01:45.80697098 +0000 UTC m=+516.159406046"
Jul 09 19:01:51.828283 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:51.828254    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-4s48q"
Jul 09 19:01:52.180511 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:52.180473    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:52.283625 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:52.283590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:58.498158 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.498123    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5789fd5768-sclqb"]
Jul 09 19:01:58.501749 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.501729    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:58.505820 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.505796    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 19:01:58.506247 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.506226    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 19:01:58.506958 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.506942    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdmcj\""
Jul 09 19:01:58.509961 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.509935    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:58.519848 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.519824    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxrfd\" (UniqueName: \"kubernetes.io/projected/32cd418c-84aa-4707-ba37-b891e77d2a26-kube-api-access-fxrfd\") pod \"maas-controller-5789fd5768-sclqb\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") " pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:58.519977 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.519881    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert\") pod \"maas-controller-5789fd5768-sclqb\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") " pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:58.520088 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.520070    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5789fd5768-sclqb"]
Jul 09 19:01:58.596657 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.596623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:58.621264 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.621234    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert\") pod \"maas-controller-5789fd5768-sclqb\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") " pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:58.621472 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.621325    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxrfd\" (UniqueName: \"kubernetes.io/projected/32cd418c-84aa-4707-ba37-b891e77d2a26-kube-api-access-fxrfd\") pod \"maas-controller-5789fd5768-sclqb\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") " pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:58.621472 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:01:58.621400    2556 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Jul 09 19:01:58.621472 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:01:58.621463    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert podName:32cd418c-84aa-4707-ba37-b891e77d2a26 nodeName:}" failed. No retries permitted until 2026-07-09 19:01:59.121444868 +0000 UTC m=+529.473879916 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert") pod "maas-controller-5789fd5768-sclqb" (UID: "32cd418c-84aa-4707-ba37-b891e77d2a26") : secret "maas-controller-webhook-cert" not found
Jul 09 19:01:58.633291 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.633264    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxrfd\" (UniqueName: \"kubernetes.io/projected/32cd418c-84aa-4707-ba37-b891e77d2a26-kube-api-access-fxrfd\") pod \"maas-controller-5789fd5768-sclqb\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") " pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:58.715145 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.715115    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:01:58.931849 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.931812    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5495d599d9-k82xv"]
Jul 09 19:01:58.935591 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.935568    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:58.944413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:58.944388    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5495d599d9-k82xv"]
Jul 09 19:01:59.025729 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.025691    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vbg5s\" (UniqueName: \"kubernetes.io/projected/1b202f15-e171-4ebf-9db2-c54253c761a5-kube-api-access-vbg5s\") pod \"maas-controller-5495d599d9-k82xv\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") " pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:59.025922 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.025755    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b202f15-e171-4ebf-9db2-c54253c761a5-webhook-cert\") pod \"maas-controller-5495d599d9-k82xv\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") " pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:59.126773 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.126733    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vbg5s\" (UniqueName: \"kubernetes.io/projected/1b202f15-e171-4ebf-9db2-c54253c761a5-kube-api-access-vbg5s\") pod \"maas-controller-5495d599d9-k82xv\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") " pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:59.126979 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.126831    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b202f15-e171-4ebf-9db2-c54253c761a5-webhook-cert\") pod \"maas-controller-5495d599d9-k82xv\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") " pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:59.126979 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.126912    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert\") pod \"maas-controller-5789fd5768-sclqb\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") " pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:59.129371 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.129347    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert\") pod \"maas-controller-5789fd5768-sclqb\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") " pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:59.129371 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.129362    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b202f15-e171-4ebf-9db2-c54253c761a5-webhook-cert\") pod \"maas-controller-5495d599d9-k82xv\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") " pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:59.138013 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.137984    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vbg5s\" (UniqueName: \"kubernetes.io/projected/1b202f15-e171-4ebf-9db2-c54253c761a5-kube-api-access-vbg5s\") pod \"maas-controller-5495d599d9-k82xv\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") " pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:59.247046 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.246961    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:01:59.386432 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.386407    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5495d599d9-k82xv"]
Jul 09 19:01:59.388897 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:01:59.388872    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b202f15_e171_4ebf_9db2_c54253c761a5.slice/crio-6604b64a65e505e8c63477d76c4fb2c78b1f0b6768eb4c36e9b05088e2536d80 WatchSource:0}: Error finding container 6604b64a65e505e8c63477d76c4fb2c78b1f0b6768eb4c36e9b05088e2536d80: Status 404 returned error can't find the container with id 6604b64a65e505e8c63477d76c4fb2c78b1f0b6768eb4c36e9b05088e2536d80
Jul 09 19:01:59.413514 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.413478    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:01:59.562630 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.562606    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5789fd5768-sclqb"]
Jul 09 19:01:59.563908 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:01:59.563880    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32cd418c_84aa_4707_ba37_b891e77d2a26.slice/crio-c72cc1438e20c3dddd7546ed394ee46fe7fa59c37de9168e61a6286a267d1b8c WatchSource:0}: Error finding container c72cc1438e20c3dddd7546ed394ee46fe7fa59c37de9168e61a6286a267d1b8c: Status 404 returned error can't find the container with id c72cc1438e20c3dddd7546ed394ee46fe7fa59c37de9168e61a6286a267d1b8c
Jul 09 19:01:59.864973 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.864878    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5495d599d9-k82xv" event={"ID":"1b202f15-e171-4ebf-9db2-c54253c761a5","Type":"ContainerStarted","Data":"6604b64a65e505e8c63477d76c4fb2c78b1f0b6768eb4c36e9b05088e2536d80"}
Jul 09 19:01:59.865974 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:01:59.865951    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5789fd5768-sclqb" event={"ID":"32cd418c-84aa-4707-ba37-b891e77d2a26","Type":"ContainerStarted","Data":"c72cc1438e20c3dddd7546ed394ee46fe7fa59c37de9168e61a6286a267d1b8c"}
Jul 09 19:02:02.651040 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.650956    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:02.830152 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.830076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:02.886326 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.886286    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5789fd5768-sclqb" event={"ID":"32cd418c-84aa-4707-ba37-b891e77d2a26","Type":"ContainerStarted","Data":"7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71"}
Jul 09 19:02:02.886570 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.886378    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:02:02.887753 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.887723    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5495d599d9-k82xv" event={"ID":"1b202f15-e171-4ebf-9db2-c54253c761a5","Type":"ContainerStarted","Data":"baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595"}
Jul 09 19:02:02.887870 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.887849    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:02:02.923831 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.923774    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5789fd5768-sclqb" podStartSLOduration=2.19009611 podStartE2EDuration="4.923757462s" totalImagesPullingTime="2.733661352s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:02.921499457 +0000 UTC m=+533.273934522" watchObservedRunningTime="2026-07-09 19:02:02.923757462 +0000 UTC m=+533.276192526"
Jul 09 19:02:02.961453 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:02.961397    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5495d599d9-k82xv" podStartSLOduration=2.05643118 podStartE2EDuration="4.96137894s" totalImagesPullingTime="2.90494776s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:01:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:02.95941727 +0000 UTC m=+533.311852335" watchObservedRunningTime="2026-07-09 19:02:02.96137894 +0000 UTC m=+533.313814036"
Jul 09 19:02:06.614982 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:06.614941    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:13.899114 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:13.899084    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:02:13.899501 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:13.899156    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:02:13.968385 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:13.968352    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5789fd5768-sclqb"]
Jul 09 19:02:13.968591 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:13.968568    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5789fd5768-sclqb" podUID="32cd418c-84aa-4707-ba37-b891e77d2a26" containerName="manager" containerID="cri-o://7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71" gracePeriod=30
Jul 09 19:02:14.235977 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.235953    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:02:14.369930 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.369894    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert\") pod \"32cd418c-84aa-4707-ba37-b891e77d2a26\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") "
Jul 09 19:02:14.370127 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.370016    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fxrfd\" (UniqueName: \"kubernetes.io/projected/32cd418c-84aa-4707-ba37-b891e77d2a26-kube-api-access-fxrfd\") pod \"32cd418c-84aa-4707-ba37-b891e77d2a26\" (UID: \"32cd418c-84aa-4707-ba37-b891e77d2a26\") "
Jul 09 19:02:14.372120 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.372091    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32cd418c-84aa-4707-ba37-b891e77d2a26-kube-api-access-fxrfd" (OuterVolumeSpecName: "kube-api-access-fxrfd") pod "32cd418c-84aa-4707-ba37-b891e77d2a26" (UID: "32cd418c-84aa-4707-ba37-b891e77d2a26"). InnerVolumeSpecName "kube-api-access-fxrfd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:02:14.372230 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.372118    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "32cd418c-84aa-4707-ba37-b891e77d2a26" (UID: "32cd418c-84aa-4707-ba37-b891e77d2a26"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:02:14.471638 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.471519    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxrfd\" (UniqueName: \"kubernetes.io/projected/32cd418c-84aa-4707-ba37-b891e77d2a26-kube-api-access-fxrfd\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:02:14.471638 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.471580    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/32cd418c-84aa-4707-ba37-b891e77d2a26-webhook-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:02:14.949501 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.949469    2556 generic.go:356] "Generic (PLEG): container finished" podID="32cd418c-84aa-4707-ba37-b891e77d2a26" containerID="7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71" exitCode=0
Jul 09 19:02:14.949977 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.949547    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5789fd5768-sclqb"
Jul 09 19:02:14.949977 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.949557    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5789fd5768-sclqb" event={"ID":"32cd418c-84aa-4707-ba37-b891e77d2a26","Type":"ContainerDied","Data":"7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71"}
Jul 09 19:02:14.949977 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.949660    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5789fd5768-sclqb" event={"ID":"32cd418c-84aa-4707-ba37-b891e77d2a26","Type":"ContainerDied","Data":"c72cc1438e20c3dddd7546ed394ee46fe7fa59c37de9168e61a6286a267d1b8c"}
Jul 09 19:02:14.949977 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.949694    2556 scope.go:117] "RemoveContainer" containerID="7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71"
Jul 09 19:02:14.958912 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.958895    2556 scope.go:117] "RemoveContainer" containerID="7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71"
Jul 09 19:02:14.959179 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:02:14.959161    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71\": container with ID starting with 7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71 not found: ID does not exist" containerID="7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71"
Jul 09 19:02:14.959217 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.959189    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71"} err="failed to get container status \"7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71\": rpc error: code = NotFound desc = could not find container \"7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71\": container with ID starting with 7970c2b43b46c476228dab10b1db0c508f821f0ca4ae1da153c545b45af4ac71 not found: ID does not exist"
Jul 09 19:02:14.984308 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.984284    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5789fd5768-sclqb"]
Jul 09 19:02:14.989644 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:14.989623    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5789fd5768-sclqb"]
Jul 09 19:02:16.084681 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:16.084646    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32cd418c-84aa-4707-ba37-b891e77d2a26" path="/var/lib/kubelet/pods/32cd418c-84aa-4707-ba37-b891e77d2a26/volumes"
Jul 09 19:02:35.928452 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:35.928410    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:36.394474 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.394387    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:36.454911 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.454880    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"]
Jul 09 19:02:36.455290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.455278    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32cd418c-84aa-4707-ba37-b891e77d2a26" containerName="manager"
Jul 09 19:02:36.455338 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.455291    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="32cd418c-84aa-4707-ba37-b891e77d2a26" containerName="manager"
Jul 09 19:02:36.455377 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.455368    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="32cd418c-84aa-4707-ba37-b891e77d2a26" containerName="manager"
Jul 09 19:02:36.460726 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.460709    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.464096 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.464067    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ktwt4\""
Jul 09 19:02:36.464208 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.464123    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 09 19:02:36.472128 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.472103    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"]
Jul 09 19:02:36.573045 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.573007    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xgqc8\" (UniqueName: \"kubernetes.io/projected/d92bbf26-9bab-451c-b5b0-65262d2ffbca-kube-api-access-xgqc8\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.573245 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.573085    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d92bbf26-9bab-451c-b5b0-65262d2ffbca-maas-api-tls\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.573245 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.573127    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92bbf26-9bab-451c-b5b0-65262d2ffbca-openshift-service-ca\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.674422 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.674390    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92bbf26-9bab-451c-b5b0-65262d2ffbca-openshift-service-ca\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.674651 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.674507    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xgqc8\" (UniqueName: \"kubernetes.io/projected/d92bbf26-9bab-451c-b5b0-65262d2ffbca-kube-api-access-xgqc8\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.674651 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.674589    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d92bbf26-9bab-451c-b5b0-65262d2ffbca-maas-api-tls\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.675150 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.675126    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d92bbf26-9bab-451c-b5b0-65262d2ffbca-openshift-service-ca\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.677138 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.677112    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d92bbf26-9bab-451c-b5b0-65262d2ffbca-maas-api-tls\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.696348 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.696319    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xgqc8\" (UniqueName: \"kubernetes.io/projected/d92bbf26-9bab-451c-b5b0-65262d2ffbca-kube-api-access-xgqc8\") pod \"maas-api-5d69b8c867-v968d\" (UID: \"d92bbf26-9bab-451c-b5b0-65262d2ffbca\") " pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.701620 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.701582    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:36.772120 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.772088    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:36.888196 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.888109    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:36.945443 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.945397    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"]
Jul 09 19:02:36.947894 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:02:36.947863    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd92bbf26_9bab_451c_b5b0_65262d2ffbca.slice/crio-a08ff5befefeaaedb4c58d21740380bbd6ba29215ab16e11e3bad7bcaac344ac WatchSource:0}: Error finding container a08ff5befefeaaedb4c58d21740380bbd6ba29215ab16e11e3bad7bcaac344ac: Status 404 returned error can't find the container with id a08ff5befefeaaedb4c58d21740380bbd6ba29215ab16e11e3bad7bcaac344ac
Jul 09 19:02:36.986933 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:36.986879    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:37.067835 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.067804    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d" event={"ID":"d92bbf26-9bab-451c-b5b0-65262d2ffbca","Type":"ContainerStarted","Data":"a08ff5befefeaaedb4c58d21740380bbd6ba29215ab16e11e3bad7bcaac344ac"}
Jul 09 19:02:37.212410 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.212327    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vzzxw"]
Jul 09 19:02:37.216849 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.216827    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.220416 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.220389    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 09 19:02:37.220580 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.220510    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-rmfgg\""
Jul 09 19:02:37.240382 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.240352    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vzzxw"]
Jul 09 19:02:37.382339 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.382305    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150574cd-7333-49b7-b727-782f54561744-plugins-config-volume\") pod \"payload-processing-58f86c7798-vzzxw\" (UID: \"150574cd-7333-49b7-b727-782f54561744\") " pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.382339 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.382342    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zggfr\" (UniqueName: \"kubernetes.io/projected/150574cd-7333-49b7-b727-782f54561744-kube-api-access-zggfr\") pod \"payload-processing-58f86c7798-vzzxw\" (UID: \"150574cd-7333-49b7-b727-782f54561744\") " pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.483046 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.482948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150574cd-7333-49b7-b727-782f54561744-plugins-config-volume\") pod \"payload-processing-58f86c7798-vzzxw\" (UID: \"150574cd-7333-49b7-b727-782f54561744\") " pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.483046 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.482987    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zggfr\" (UniqueName: \"kubernetes.io/projected/150574cd-7333-49b7-b727-782f54561744-kube-api-access-zggfr\") pod \"payload-processing-58f86c7798-vzzxw\" (UID: \"150574cd-7333-49b7-b727-782f54561744\") " pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.483692 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.483668    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/150574cd-7333-49b7-b727-782f54561744-plugins-config-volume\") pod \"payload-processing-58f86c7798-vzzxw\" (UID: \"150574cd-7333-49b7-b727-782f54561744\") " pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.497304 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.497278    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zggfr\" (UniqueName: \"kubernetes.io/projected/150574cd-7333-49b7-b727-782f54561744-kube-api-access-zggfr\") pod \"payload-processing-58f86c7798-vzzxw\" (UID: \"150574cd-7333-49b7-b727-782f54561744\") " pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.528798 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.528768    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:37.666623 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:37.666594    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-58f86c7798-vzzxw"]
Jul 09 19:02:37.668501 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:02:37.668479    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod150574cd_7333_49b7_b727_782f54561744.slice/crio-411747d5625d4c994076720757b3c7dbbe4f40a83d2580e5f41856e7af2a162b WatchSource:0}: Error finding container 411747d5625d4c994076720757b3c7dbbe4f40a83d2580e5f41856e7af2a162b: Status 404 returned error can't find the container with id 411747d5625d4c994076720757b3c7dbbe4f40a83d2580e5f41856e7af2a162b
Jul 09 19:02:38.075907 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:38.075865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-vzzxw" event={"ID":"150574cd-7333-49b7-b727-782f54561744","Type":"ContainerStarted","Data":"411747d5625d4c994076720757b3c7dbbe4f40a83d2580e5f41856e7af2a162b"}
Jul 09 19:02:40.090058 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:40.090027    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-58f86c7798-vzzxw" event={"ID":"150574cd-7333-49b7-b727-782f54561744","Type":"ContainerStarted","Data":"a94f0090ac651d7e40cb561e528a38d7e87149da1073ea610d638e50b386ee98"}
Jul 09 19:02:40.090486 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:40.090139    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:40.091673 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:40.091643    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d" event={"ID":"d92bbf26-9bab-451c-b5b0-65262d2ffbca","Type":"ContainerStarted","Data":"84144c770728b177d3e8ba07e3d1f491475c71fc294d56672e8400542f3f79ce"}
Jul 09 19:02:40.091779 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:40.091718    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:40.106051 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:40.105989    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-58f86c7798-vzzxw" podStartSLOduration=1.538981833 podStartE2EDuration="3.10596722s" totalImagesPullingTime="1.566985387s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:40.105126102 +0000 UTC m=+570.457561167" watchObservedRunningTime="2026-07-09 19:02:40.10596722 +0000 UTC m=+570.458402282"
Jul 09 19:02:40.123317 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:40.123268    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d" podStartSLOduration=1.837059126 podStartE2EDuration="4.123250728s" totalImagesPullingTime="2.286191602s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:02:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:02:40.122406305 +0000 UTC m=+570.474841370" watchObservedRunningTime="2026-07-09 19:02:40.123250728 +0000 UTC m=+570.475685792"
Jul 09 19:02:46.107031 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:46.107002    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5d69b8c867-v968d"
Jul 09 19:02:47.166653 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:47.166617    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:47.347089 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:47.347045    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:51.102840 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:51.102811    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-58f86c7798-vzzxw"
Jul 09 19:02:55.474065 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:55.474029    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:55.574694 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:55.574664    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:56.479311 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:56.479276    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:02:56.531021 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:02:56.530984    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:04.395972 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:04.395937    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:10.024431 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:10.024403    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:03:10.025153 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:10.025134    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:03:10.029717 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:10.029696    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:03:10.030336 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:10.030319    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:03:14.935500 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.935467    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"]
Jul 09 19:03:14.938962 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.938940    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:14.941694 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.941675    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-l6hgx\""
Jul 09 19:03:14.941866 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.941839    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 09 19:03:14.942760 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.942743    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 09 19:03:14.942867 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.942759    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 09 19:03:14.948592 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.948565    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"]
Jul 09 19:03:14.973262 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:14.973224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:15.013702 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.013668    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.013884 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.013715    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6q9\" (UniqueName: \"kubernetes.io/projected/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-kube-api-access-qz6q9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.013884 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.013810    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.013884 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.013846    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.014041 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.013925    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.014041 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.013962    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115388 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115353    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115585 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115398    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115585 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115552    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115718 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115608    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6q9\" (UniqueName: \"kubernetes.io/projected/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-kube-api-access-qz6q9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115718 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115669    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115718 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115684    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115866 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115814    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115918 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115888    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.115974 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.115956    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.117767 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.117734    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.117940 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.117923    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.123417 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.123391    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6q9\" (UniqueName: \"kubernetes.io/projected/e4d9e3c4-5595-4828-8374-154e6bcf2a1c-kube-api-access-qz6q9\") pod \"e2e-distinct-simulated-kserve-59975d69f9-2xmnr\" (UID: \"e4d9e3c4-5595-4828-8374-154e6bcf2a1c\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.250488 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.250391    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:15.404466 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.404429    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"]
Jul 09 19:03:15.407833 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:03:15.407805    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4d9e3c4_5595_4828_8374_154e6bcf2a1c.slice/crio-0fbf9862c34a039aae503d11b1254ea4604bfd0482ee34c4b4595d7719c4dabc WatchSource:0}: Error finding container 0fbf9862c34a039aae503d11b1254ea4604bfd0482ee34c4b4595d7719c4dabc: Status 404 returned error can't find the container with id 0fbf9862c34a039aae503d11b1254ea4604bfd0482ee34c4b4595d7719c4dabc
Jul 09 19:03:15.409324 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:15.409304    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:03:16.281214 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:16.281180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr" event={"ID":"e4d9e3c4-5595-4828-8374-154e6bcf2a1c","Type":"ContainerStarted","Data":"0fbf9862c34a039aae503d11b1254ea4604bfd0482ee34c4b4595d7719c4dabc"}
Jul 09 19:03:17.289024 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:17.288987    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr" event={"ID":"e4d9e3c4-5595-4828-8374-154e6bcf2a1c","Type":"ContainerStarted","Data":"3bd712cea7c9282047a8a098325ba53a26f999b3fbcd9b25f0f71ca4581c36e7"}
Jul 09 19:03:17.340075 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:17.340021    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr" podStartSLOduration=2.224192501 podStartE2EDuration="3.340003444s" totalImagesPullingTime="1.115810943s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:03:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:03:17.338637223 +0000 UTC m=+607.691072287" watchObservedRunningTime="2026-07-09 19:03:17.340003444 +0000 UTC m=+607.692438509"
Jul 09 19:03:25.251297 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:25.251253    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:25.251748 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:25.251591    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:25.263994 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:25.263969    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:25.350411 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:25.350381    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-2xmnr"
Jul 09 19:03:37.708025 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:37.707940    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:40.379792 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:40.379752    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:43.060100 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:43.060063    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:45.818046 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:45.818014    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:03:55.886255 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:03:55.886207    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:12.095749 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:04:12.095707    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:43.349404 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:04:43.349366    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:43.451814 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:04:43.451776    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:44.472398 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:04:44.472365    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:04:44.587444 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:04:44.587413    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:00.486838 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:00.486795    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:00.587177 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:00.587131    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:03.915034 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:03.915004    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:07.034748 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:07.034712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:07.139808 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:07.139775    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:14.525016 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:14.524985    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:23.808581 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:23.808509    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:55.719061 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:55.719027    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:55.835695 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:55.835661    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5495d599d9-k82xv"]
Jul 09 19:05:55.835948 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:55.835924    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-5495d599d9-k82xv" podUID="1b202f15-e171-4ebf-9db2-c54253c761a5" containerName="manager" containerID="cri-o://baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595" gracePeriod=30
Jul 09 19:05:56.102843 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.102812    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:05:56.123510 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.123469    2556 generic.go:356] "Generic (PLEG): container finished" podID="1b202f15-e171-4ebf-9db2-c54253c761a5" containerID="baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595" exitCode=0
Jul 09 19:05:56.123691 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.123552    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5495d599d9-k82xv"
Jul 09 19:05:56.123691 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.123562    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5495d599d9-k82xv" event={"ID":"1b202f15-e171-4ebf-9db2-c54253c761a5","Type":"ContainerDied","Data":"baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595"}
Jul 09 19:05:56.123691 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.123621    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5495d599d9-k82xv" event={"ID":"1b202f15-e171-4ebf-9db2-c54253c761a5","Type":"ContainerDied","Data":"6604b64a65e505e8c63477d76c4fb2c78b1f0b6768eb4c36e9b05088e2536d80"}
Jul 09 19:05:56.123691 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.123651    2556 scope.go:117] "RemoveContainer" containerID="baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595"
Jul 09 19:05:56.135272 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.135251    2556 scope.go:117] "RemoveContainer" containerID="baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595"
Jul 09 19:05:56.135661 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:05:56.135628    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595\": container with ID starting with baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595 not found: ID does not exist" containerID="baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595"
Jul 09 19:05:56.135772 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.135669    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595"} err="failed to get container status \"baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595\": rpc error: code = NotFound desc = could not find container \"baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595\": container with ID starting with baf624bb440ed46b363a2881a7d8445909c403c52158ce3ee9c783d728961595 not found: ID does not exist"
Jul 09 19:05:56.143076 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.143049    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vbg5s\" (UniqueName: \"kubernetes.io/projected/1b202f15-e171-4ebf-9db2-c54253c761a5-kube-api-access-vbg5s\") pod \"1b202f15-e171-4ebf-9db2-c54253c761a5\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") "
Jul 09 19:05:56.143205 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.143113    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b202f15-e171-4ebf-9db2-c54253c761a5-webhook-cert\") pod \"1b202f15-e171-4ebf-9db2-c54253c761a5\" (UID: \"1b202f15-e171-4ebf-9db2-c54253c761a5\") "
Jul 09 19:05:56.145753 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.145718    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1b202f15-e171-4ebf-9db2-c54253c761a5-kube-api-access-vbg5s" (OuterVolumeSpecName: "kube-api-access-vbg5s") pod "1b202f15-e171-4ebf-9db2-c54253c761a5" (UID: "1b202f15-e171-4ebf-9db2-c54253c761a5"). InnerVolumeSpecName "kube-api-access-vbg5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:05:56.145865 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.145793    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1b202f15-e171-4ebf-9db2-c54253c761a5-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "1b202f15-e171-4ebf-9db2-c54253c761a5" (UID: "1b202f15-e171-4ebf-9db2-c54253c761a5"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:05:56.244107 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.244065    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vbg5s\" (UniqueName: \"kubernetes.io/projected/1b202f15-e171-4ebf-9db2-c54253c761a5-kube-api-access-vbg5s\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:05:56.244107 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.244098    2556 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1b202f15-e171-4ebf-9db2-c54253c761a5-webhook-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:05:56.446615 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.446583    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-5495d599d9-k82xv"]
Jul 09 19:05:56.450484 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:56.450456    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-5495d599d9-k82xv"]
Jul 09 19:05:57.118951 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.118915    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:05:57.236478 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.236440    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-5495d599d9-ht6jg"]
Jul 09 19:05:57.237000 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.236980    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1b202f15-e171-4ebf-9db2-c54253c761a5" containerName="manager"
Jul 09 19:05:57.237000 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.237002    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1b202f15-e171-4ebf-9db2-c54253c761a5" containerName="manager"
Jul 09 19:05:57.237164 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.237149    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1b202f15-e171-4ebf-9db2-c54253c761a5" containerName="manager"
Jul 09 19:05:57.241619 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.241595    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.245445 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.245416    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-hdmcj\""
Jul 09 19:05:57.245445 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.245434    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 09 19:05:57.245445 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.245416    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 09 19:05:57.250111 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.250089    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5495d599d9-ht6jg"]
Jul 09 19:05:57.353141 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.353100    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2791912b-a71c-4636-8269-ca88a46628da-webhook-cert\") pod \"maas-controller-5495d599d9-ht6jg\" (UID: \"2791912b-a71c-4636-8269-ca88a46628da\") " pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.353334 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.353226    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w8kl4\" (UniqueName: \"kubernetes.io/projected/2791912b-a71c-4636-8269-ca88a46628da-kube-api-access-w8kl4\") pod \"maas-controller-5495d599d9-ht6jg\" (UID: \"2791912b-a71c-4636-8269-ca88a46628da\") " pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.454372 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.454335    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w8kl4\" (UniqueName: \"kubernetes.io/projected/2791912b-a71c-4636-8269-ca88a46628da-kube-api-access-w8kl4\") pod \"maas-controller-5495d599d9-ht6jg\" (UID: \"2791912b-a71c-4636-8269-ca88a46628da\") " pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.454623 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.454409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2791912b-a71c-4636-8269-ca88a46628da-webhook-cert\") pod \"maas-controller-5495d599d9-ht6jg\" (UID: \"2791912b-a71c-4636-8269-ca88a46628da\") " pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.456861 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.456841    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2791912b-a71c-4636-8269-ca88a46628da-webhook-cert\") pod \"maas-controller-5495d599d9-ht6jg\" (UID: \"2791912b-a71c-4636-8269-ca88a46628da\") " pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.462964 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.462934    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w8kl4\" (UniqueName: \"kubernetes.io/projected/2791912b-a71c-4636-8269-ca88a46628da-kube-api-access-w8kl4\") pod \"maas-controller-5495d599d9-ht6jg\" (UID: \"2791912b-a71c-4636-8269-ca88a46628da\") " pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.552917 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.552878    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:57.694401 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:57.694373    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-5495d599d9-ht6jg"]
Jul 09 19:05:57.696792 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:05:57.696765    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2791912b_a71c_4636_8269_ca88a46628da.slice/crio-1676cfeddc238631ac503baf0790a07ea85c06e169b5175bcb4df16c5acbe04b WatchSource:0}: Error finding container 1676cfeddc238631ac503baf0790a07ea85c06e169b5175bcb4df16c5acbe04b: Status 404 returned error can't find the container with id 1676cfeddc238631ac503baf0790a07ea85c06e169b5175bcb4df16c5acbe04b
Jul 09 19:05:58.087023 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:58.086985    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1b202f15-e171-4ebf-9db2-c54253c761a5" path="/var/lib/kubelet/pods/1b202f15-e171-4ebf-9db2-c54253c761a5/volumes"
Jul 09 19:05:58.136679 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:58.136643    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5495d599d9-ht6jg" event={"ID":"2791912b-a71c-4636-8269-ca88a46628da","Type":"ContainerStarted","Data":"1676cfeddc238631ac503baf0790a07ea85c06e169b5175bcb4df16c5acbe04b"}
Jul 09 19:05:59.142826 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:59.142791    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-5495d599d9-ht6jg" event={"ID":"2791912b-a71c-4636-8269-ca88a46628da","Type":"ContainerStarted","Data":"6e3accf4b5610a3d17a82d459bc3aae0c5470f67fa2a69a25c9bc9d4fd7c25bc"}
Jul 09 19:05:59.143211 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:59.142897    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:05:59.161413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:05:59.161359    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-5495d599d9-ht6jg" podStartSLOduration=1.4048497690000001 podStartE2EDuration="2.161340435s" totalImagesPullingTime="756.490666ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:05:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:05:59.160271445 +0000 UTC m=+769.512706511" watchObservedRunningTime="2026-07-09 19:05:59.161340435 +0000 UTC m=+769.513775549"
Jul 09 19:06:10.153910 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:10.153872    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-5495d599d9-ht6jg"
Jul 09 19:06:20.894269 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:20.894231    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:40.613272 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:40.613178    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:40.714305 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:40.714267    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:40.841399 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:40.841363    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:46.966905 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:46.966862    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:47.081368 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:47.081324    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:47.216337 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:47.216300    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:53.035381 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:53.035346    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:06:53.139054 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:06:53.139018    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:08:10.067868 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:08:10.067788    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:08:10.068446 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:08:10.068094    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:08:10.074978 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:08:10.074956    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:08:10.075137 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:08:10.075041    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:10:46.627453 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:46.627412    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"]
Jul 09 19:10:46.628078 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:46.627675    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv" podUID="156eb454-8dfa-4859-b6ba-0134e38cc87b" containerName="manager" containerID="cri-o://448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf" gracePeriod=10
Jul 09 19:10:51.093518 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.093490    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:10:51.138854 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.138762    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/156eb454-8dfa-4859-b6ba-0134e38cc87b-extensions-socket-volume\") pod \"156eb454-8dfa-4859-b6ba-0134e38cc87b\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") "
Jul 09 19:10:51.138854 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.138814    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/156eb454-8dfa-4859-b6ba-0134e38cc87b-kube-api-access-mds7f\") pod \"156eb454-8dfa-4859-b6ba-0134e38cc87b\" (UID: \"156eb454-8dfa-4859-b6ba-0134e38cc87b\") "
Jul 09 19:10:51.139170 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.139146    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/156eb454-8dfa-4859-b6ba-0134e38cc87b-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "156eb454-8dfa-4859-b6ba-0134e38cc87b" (UID: "156eb454-8dfa-4859-b6ba-0134e38cc87b"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:10:51.140990 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.140956    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/156eb454-8dfa-4859-b6ba-0134e38cc87b-kube-api-access-mds7f" (OuterVolumeSpecName: "kube-api-access-mds7f") pod "156eb454-8dfa-4859-b6ba-0134e38cc87b" (UID: "156eb454-8dfa-4859-b6ba-0134e38cc87b"). InnerVolumeSpecName "kube-api-access-mds7f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:10:51.239638 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.239600    2556 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/156eb454-8dfa-4859-b6ba-0134e38cc87b-extensions-socket-volume\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:10:51.239638 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.239632    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mds7f\" (UniqueName: \"kubernetes.io/projected/156eb454-8dfa-4859-b6ba-0134e38cc87b-kube-api-access-mds7f\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:10:51.659671 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.659638    2556 generic.go:356] "Generic (PLEG): container finished" podID="156eb454-8dfa-4859-b6ba-0134e38cc87b" containerID="448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf" exitCode=0
Jul 09 19:10:51.659921 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.659680    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"
Jul 09 19:10:51.659921 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.659696    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv" event={"ID":"156eb454-8dfa-4859-b6ba-0134e38cc87b","Type":"ContainerDied","Data":"448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf"}
Jul 09 19:10:51.659921 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.659751    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv" event={"ID":"156eb454-8dfa-4859-b6ba-0134e38cc87b","Type":"ContainerDied","Data":"55319d2d3b594551b85a274305f42d199e4694b79e65127bfe73c3f0223a07fc"}
Jul 09 19:10:51.659921 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.659774    2556 scope.go:117] "RemoveContainer" containerID="448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf"
Jul 09 19:10:51.670169 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.670150    2556 scope.go:117] "RemoveContainer" containerID="448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf"
Jul 09 19:10:51.670476 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:10:51.670458    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf\": container with ID starting with 448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf not found: ID does not exist" containerID="448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf"
Jul 09 19:10:51.670564 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.670490    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf"} err="failed to get container status \"448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf\": rpc error: code = NotFound desc = could not find container \"448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf\": container with ID starting with 448c5f0c30ada71ecc98bbf5909e1287e29a2464d101b0ecbbcddbfe487950cf not found: ID does not exist"
Jul 09 19:10:51.682889 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.682859    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"]
Jul 09 19:10:51.686748 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:51.686724    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-hwrxv"]
Jul 09 19:10:52.084998 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:10:52.084961    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="156eb454-8dfa-4859-b6ba-0134e38cc87b" path="/var/lib/kubelet/pods/156eb454-8dfa-4859-b6ba-0134e38cc87b/volumes"
Jul 09 19:11:35.604799 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:35.604752    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:39.859694 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:39.859649    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:47.799469 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.799437    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"]
Jul 09 19:11:47.799941 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.799859    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="156eb454-8dfa-4859-b6ba-0134e38cc87b" containerName="manager"
Jul 09 19:11:47.799941 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.799870    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="156eb454-8dfa-4859-b6ba-0134e38cc87b" containerName="manager"
Jul 09 19:11:47.800036 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.799952    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="156eb454-8dfa-4859-b6ba-0134e38cc87b" containerName="manager"
Jul 09 19:11:47.803055 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.803039    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:47.805878 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.805857    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-zq2wt\""
Jul 09 19:11:47.815517 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.815495    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"]
Jul 09 19:11:47.846056 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.846028    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c337e848-7ba3-423e-9bc2-80b3fc391ae1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c694q\" (UID: \"c337e848-7ba3-423e-9bc2-80b3fc391ae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:47.846220 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.846167    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mbthx\" (UniqueName: \"kubernetes.io/projected/c337e848-7ba3-423e-9bc2-80b3fc391ae1-kube-api-access-mbthx\") pod \"kuadrant-operator-controller-manager-86878c769f-c694q\" (UID: \"c337e848-7ba3-423e-9bc2-80b3fc391ae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:47.946810 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.946772    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mbthx\" (UniqueName: \"kubernetes.io/projected/c337e848-7ba3-423e-9bc2-80b3fc391ae1-kube-api-access-mbthx\") pod \"kuadrant-operator-controller-manager-86878c769f-c694q\" (UID: \"c337e848-7ba3-423e-9bc2-80b3fc391ae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:47.946976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.946828    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c337e848-7ba3-423e-9bc2-80b3fc391ae1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c694q\" (UID: \"c337e848-7ba3-423e-9bc2-80b3fc391ae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:47.947242 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.947222    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/c337e848-7ba3-423e-9bc2-80b3fc391ae1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-c694q\" (UID: \"c337e848-7ba3-423e-9bc2-80b3fc391ae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:47.955543 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:47.955512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mbthx\" (UniqueName: \"kubernetes.io/projected/c337e848-7ba3-423e-9bc2-80b3fc391ae1-kube-api-access-mbthx\") pod \"kuadrant-operator-controller-manager-86878c769f-c694q\" (UID: \"c337e848-7ba3-423e-9bc2-80b3fc391ae1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:48.113600 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:48.113496    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:48.259032 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:48.259000    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"]
Jul 09 19:11:48.263292 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:11:48.263267    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc337e848_7ba3_423e_9bc2_80b3fc391ae1.slice/crio-db98bf4f6742852069d140bae5bdd7665bc220d9aab8542168d1f5ae417036ab WatchSource:0}: Error finding container db98bf4f6742852069d140bae5bdd7665bc220d9aab8542168d1f5ae417036ab: Status 404 returned error can't find the container with id db98bf4f6742852069d140bae5bdd7665bc220d9aab8542168d1f5ae417036ab
Jul 09 19:11:48.955732 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:48.955702    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q" event={"ID":"c337e848-7ba3-423e-9bc2-80b3fc391ae1","Type":"ContainerStarted","Data":"bf979eeececa34c874908b02c6bafa2c6c050b74f0436d2aeab1de4e1a2f6911"}
Jul 09 19:11:48.956115 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:48.955750    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q" event={"ID":"c337e848-7ba3-423e-9bc2-80b3fc391ae1","Type":"ContainerStarted","Data":"db98bf4f6742852069d140bae5bdd7665bc220d9aab8542168d1f5ae417036ab"}
Jul 09 19:11:48.956115 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:48.955824    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:11:48.978476 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:48.978420    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q" podStartSLOduration=1.9784007190000001 podStartE2EDuration="1.978400719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:11:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:11:48.976785951 +0000 UTC m=+1119.329221019" watchObservedRunningTime="2026-07-09 19:11:48.978400719 +0000 UTC m=+1119.330835786"
Jul 09 19:11:58.863710 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:58.863680    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:58.964043 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:58.964008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:11:59.962927 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:11:59.962895    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-c694q"
Jul 09 19:13:10.106317 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:13:10.106293    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:13:10.107282 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:13:10.107263    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:13:10.111457 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:13:10.111440    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:13:10.112579 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:13:10.112565    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:15:00.142442 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.142398    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"]
Jul 09 19:15:00.145886 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.145866    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"
Jul 09 19:15:00.166019 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.165989    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"]
Jul 09 19:15:00.214830 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.214784    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsw8r\" (UniqueName: \"kubernetes.io/projected/ae6d5a61-7010-41fc-82f9-8584559beabc-kube-api-access-xsw8r\") pod \"maas-api-key-cleanup-29727075-rglpb\" (UID: \"ae6d5a61-7010-41fc-82f9-8584559beabc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"
Jul 09 19:15:00.316190 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.316150    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsw8r\" (UniqueName: \"kubernetes.io/projected/ae6d5a61-7010-41fc-82f9-8584559beabc-kube-api-access-xsw8r\") pod \"maas-api-key-cleanup-29727075-rglpb\" (UID: \"ae6d5a61-7010-41fc-82f9-8584559beabc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"
Jul 09 19:15:00.325468 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.325440    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsw8r\" (UniqueName: \"kubernetes.io/projected/ae6d5a61-7010-41fc-82f9-8584559beabc-kube-api-access-xsw8r\") pod \"maas-api-key-cleanup-29727075-rglpb\" (UID: \"ae6d5a61-7010-41fc-82f9-8584559beabc\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"
Jul 09 19:15:00.455641 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.455605    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"
Jul 09 19:15:00.600249 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.600220    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"]
Jul 09 19:15:00.602674 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:15:00.602647    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae6d5a61_7010_41fc_82f9_8584559beabc.slice/crio-ddd3cf9442fb8e6cd4743a93679b481f53967207111a9a125341052031af42cc WatchSource:0}: Error finding container ddd3cf9442fb8e6cd4743a93679b481f53967207111a9a125341052031af42cc: Status 404 returned error can't find the container with id ddd3cf9442fb8e6cd4743a93679b481f53967207111a9a125341052031af42cc
Jul 09 19:15:00.604539 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.604507    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:15:00.946729 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:00.946695    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb" event={"ID":"ae6d5a61-7010-41fc-82f9-8584559beabc","Type":"ContainerStarted","Data":"ddd3cf9442fb8e6cd4743a93679b481f53967207111a9a125341052031af42cc"}
Jul 09 19:15:03.967479 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:03.967379    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb" event={"ID":"ae6d5a61-7010-41fc-82f9-8584559beabc","Type":"ContainerStarted","Data":"e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798"}
Jul 09 19:15:03.984411 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:15:03.984355    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb" podStartSLOduration=1.150392195 podStartE2EDuration="3.984336394s" totalImagesPullingTime="2.833944199s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:15:03.984033595 +0000 UTC m=+1314.336468660" watchObservedRunningTime="2026-07-09 19:15:03.984336394 +0000 UTC m=+1314.336771459"
Jul 09 19:16:00.018589 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:00.018548    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"]
Jul 09 19:16:00.019171 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:00.018754    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb" podUID="ae6d5a61-7010-41fc-82f9-8584559beabc" containerName="cleanup" containerID="cri-o://e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798" gracePeriod=30
Jul 09 19:16:15.976491 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:15.976445    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:16:30.215074 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.215039    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"
Jul 09 19:16:30.227971 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.227948    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsw8r\" (UniqueName: \"kubernetes.io/projected/ae6d5a61-7010-41fc-82f9-8584559beabc-kube-api-access-xsw8r\") pod \"ae6d5a61-7010-41fc-82f9-8584559beabc\" (UID: \"ae6d5a61-7010-41fc-82f9-8584559beabc\") "
Jul 09 19:16:30.230012 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.229981    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae6d5a61-7010-41fc-82f9-8584559beabc-kube-api-access-xsw8r" (OuterVolumeSpecName: "kube-api-access-xsw8r") pod "ae6d5a61-7010-41fc-82f9-8584559beabc" (UID: "ae6d5a61-7010-41fc-82f9-8584559beabc"). InnerVolumeSpecName "kube-api-access-xsw8r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:16:30.328854 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.328773    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsw8r\" (UniqueName: \"kubernetes.io/projected/ae6d5a61-7010-41fc-82f9-8584559beabc-kube-api-access-xsw8r\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:16:30.411469 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.411434    2556 generic.go:356] "Generic (PLEG): container finished" podID="ae6d5a61-7010-41fc-82f9-8584559beabc" containerID="e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798" exitCode=137
Jul 09 19:16:30.411662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.411481    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"
Jul 09 19:16:30.411662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.411491    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb" event={"ID":"ae6d5a61-7010-41fc-82f9-8584559beabc","Type":"ContainerDied","Data":"e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798"}
Jul 09 19:16:30.411662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.411555    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb" event={"ID":"ae6d5a61-7010-41fc-82f9-8584559beabc","Type":"ContainerDied","Data":"ddd3cf9442fb8e6cd4743a93679b481f53967207111a9a125341052031af42cc"}
Jul 09 19:16:30.411662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.411579    2556 scope.go:117] "RemoveContainer" containerID="e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798"
Jul 09 19:16:30.421514 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.421497    2556 scope.go:117] "RemoveContainer" containerID="e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798"
Jul 09 19:16:30.423321 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:16:30.423298    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798\": container with ID starting with e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798 not found: ID does not exist" containerID="e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798"
Jul 09 19:16:30.423413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.423328    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798"} err="failed to get container status \"e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798\": rpc error: code = NotFound desc = could not find container \"e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798\": container with ID starting with e462c2b05647c89929d790c9e21830f8e4ba4799438e21a408ad9b9e560b4798 not found: ID does not exist"
Jul 09 19:16:30.441141 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.441111    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"]
Jul 09 19:16:30.443315 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:30.443293    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727075-rglpb"]
Jul 09 19:16:32.084903 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:16:32.084870    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ae6d5a61-7010-41fc-82f9-8584559beabc" path="/var/lib/kubelet/pods/ae6d5a61-7010-41fc-82f9-8584559beabc/volumes"
Jul 09 19:17:40.273057 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:17:40.273010    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:17:46.903113 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:17:46.903076    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:17:55.162784 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:17:55.162744    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:00.297478 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:00.297429    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:03.283772 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:03.283735    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:03.393145 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:03.393114    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:03.731341 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:03.731305    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:09.614072 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:09.614033    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:09.713768 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:09.713731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:09.862964 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:09.862925    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:09.940473 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:09.940438    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:10.136903 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:10.136870    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:18:10.139653 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:10.139624    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:18:10.142113 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:10.142098    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:18:10.144796 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:10.144780    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:18:11.012567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.012487    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:11.091933 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.091899    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"]
Jul 09 19:18:11.092875 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.092475    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ae6d5a61-7010-41fc-82f9-8584559beabc" containerName="cleanup"
Jul 09 19:18:11.092875 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.092489    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="ae6d5a61-7010-41fc-82f9-8584559beabc" containerName="cleanup"
Jul 09 19:18:11.092875 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.092608    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="ae6d5a61-7010-41fc-82f9-8584559beabc" containerName="cleanup"
Jul 09 19:18:11.094987 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.094957    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.095865 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.095841    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:11.097751 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.097734    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-14bca9d0-serving-cert\""
Jul 09 19:18:11.109359 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.109333    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"]
Jul 09 19:18:11.170883 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.170846    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:11.223813 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.223783    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.224051 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.223823    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bghz5\" (UniqueName: \"kubernetes.io/projected/efaab627-e085-49d5-83ba-5d2fbda302ee-kube-api-access-bghz5\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.224051 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.224000    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efaab627-e085-49d5-83ba-5d2fbda302ee-openshift-service-ca\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.325062 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.324966    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efaab627-e085-49d5-83ba-5d2fbda302ee-openshift-service-ca\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.325062 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.325038    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.325302 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.325075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bghz5\" (UniqueName: \"kubernetes.io/projected/efaab627-e085-49d5-83ba-5d2fbda302ee-kube-api-access-bghz5\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.325302 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:18:11.325177    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-serving-cert: secret "maas-api-e2e-ait-14bca9d0-serving-cert" not found
Jul 09 19:18:11.325302 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:18:11.325254    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls podName:efaab627-e085-49d5-83ba-5d2fbda302ee nodeName:}" failed. No retries permitted until 2026-07-09 19:18:11.82523657 +0000 UTC m=+1502.177671616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls") pod "maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz" (UID: "efaab627-e085-49d5-83ba-5d2fbda302ee") : secret "maas-api-e2e-ait-14bca9d0-serving-cert" not found
Jul 09 19:18:11.325819 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.325798    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efaab627-e085-49d5-83ba-5d2fbda302ee-openshift-service-ca\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.344328 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.344299    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bghz5\" (UniqueName: \"kubernetes.io/projected/efaab627-e085-49d5-83ba-5d2fbda302ee-kube-api-access-bghz5\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.830420 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.830386    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.832922 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.832896    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls\") pod \"maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:11.856318 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.856284    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:11.954007 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:11.953969    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:12.007285 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:12.007253    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:12.054933 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:12.054895    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:12.148347 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:12.148320    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"]
Jul 09 19:18:12.150401 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:18:12.150374    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefaab627_e085_49d5_83ba_5d2fbda302ee.slice/crio-15edde7e39d08d538af19955fb7fded03d4f2430964c9add9873fdc53ae056be WatchSource:0}: Error finding container 15edde7e39d08d538af19955fb7fded03d4f2430964c9add9873fdc53ae056be: Status 404 returned error can't find the container with id 15edde7e39d08d538af19955fb7fded03d4f2430964c9add9873fdc53ae056be
Jul 09 19:18:12.950541 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:12.950438    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz" event={"ID":"efaab627-e085-49d5-83ba-5d2fbda302ee","Type":"ContainerStarted","Data":"013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273"}
Jul 09 19:18:12.950541 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:12.950491    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz" event={"ID":"efaab627-e085-49d5-83ba-5d2fbda302ee","Type":"ContainerStarted","Data":"15edde7e39d08d538af19955fb7fded03d4f2430964c9add9873fdc53ae056be"}
Jul 09 19:18:12.950739 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:12.950556    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:12.968124 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:12.968077    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz" podStartSLOduration=1.438962407 podStartE2EDuration="1.968061882s" totalImagesPullingTime="529.099475ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:18:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:18:12.966705515 +0000 UTC m=+1503.319140582" watchObservedRunningTime="2026-07-09 19:18:12.968061882 +0000 UTC m=+1503.320496947"
Jul 09 19:18:13.540228 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:13.540197    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:16.064914 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.064880    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:16.274695 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.274644    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:16.356984 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.356873    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:16.421361 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.421329    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"]
Jul 09 19:18:16.421608 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.421582    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz" podUID="efaab627-e085-49d5-83ba-5d2fbda302ee" containerName="maas-api" containerID="cri-o://013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273" gracePeriod=30
Jul 09 19:18:16.492848 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.492809    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:16.679538 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.679496    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:16.769691 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.769663    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls\") pod \"efaab627-e085-49d5-83ba-5d2fbda302ee\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") "
Jul 09 19:18:16.769899 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.769699    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bghz5\" (UniqueName: \"kubernetes.io/projected/efaab627-e085-49d5-83ba-5d2fbda302ee-kube-api-access-bghz5\") pod \"efaab627-e085-49d5-83ba-5d2fbda302ee\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") "
Jul 09 19:18:16.769899 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.769731    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efaab627-e085-49d5-83ba-5d2fbda302ee-openshift-service-ca\") pod \"efaab627-e085-49d5-83ba-5d2fbda302ee\" (UID: \"efaab627-e085-49d5-83ba-5d2fbda302ee\") "
Jul 09 19:18:16.770194 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.770164    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efaab627-e085-49d5-83ba-5d2fbda302ee-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "efaab627-e085-49d5-83ba-5d2fbda302ee" (UID: "efaab627-e085-49d5-83ba-5d2fbda302ee"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:18:16.771964 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.771939    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "efaab627-e085-49d5-83ba-5d2fbda302ee" (UID: "efaab627-e085-49d5-83ba-5d2fbda302ee"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:18:16.772070 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.771974    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efaab627-e085-49d5-83ba-5d2fbda302ee-kube-api-access-bghz5" (OuterVolumeSpecName: "kube-api-access-bghz5") pod "efaab627-e085-49d5-83ba-5d2fbda302ee" (UID: "efaab627-e085-49d5-83ba-5d2fbda302ee"). InnerVolumeSpecName "kube-api-access-bghz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:18:16.871085 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.871056    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/efaab627-e085-49d5-83ba-5d2fbda302ee-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:16.871085 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.871083    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bghz5\" (UniqueName: \"kubernetes.io/projected/efaab627-e085-49d5-83ba-5d2fbda302ee-kube-api-access-bghz5\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:16.871085 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.871093    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/efaab627-e085-49d5-83ba-5d2fbda302ee-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:16.973860 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.973823    2556 generic.go:356] "Generic (PLEG): container finished" podID="efaab627-e085-49d5-83ba-5d2fbda302ee" containerID="013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273" exitCode=0
Jul 09 19:18:16.974036 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.973880    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"
Jul 09 19:18:16.974036 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.973912    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz" event={"ID":"efaab627-e085-49d5-83ba-5d2fbda302ee","Type":"ContainerDied","Data":"013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273"}
Jul 09 19:18:16.974036 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.973977    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz" event={"ID":"efaab627-e085-49d5-83ba-5d2fbda302ee","Type":"ContainerDied","Data":"15edde7e39d08d538af19955fb7fded03d4f2430964c9add9873fdc53ae056be"}
Jul 09 19:18:16.974036 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.974011    2556 scope.go:117] "RemoveContainer" containerID="013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273"
Jul 09 19:18:16.984185 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.984163    2556 scope.go:117] "RemoveContainer" containerID="013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273"
Jul 09 19:18:16.984442 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:18:16.984424    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273\": container with ID starting with 013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273 not found: ID does not exist" containerID="013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273"
Jul 09 19:18:16.984491 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.984452    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273"} err="failed to get container status \"013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273\": rpc error: code = NotFound desc = could not find container \"013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273\": container with ID starting with 013cb9272179292938d874af92a03095480026714be3f9e686137268511a3273 not found: ID does not exist"
Jul 09 19:18:16.995693 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.995666    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"]
Jul 09 19:18:16.999452 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:16.999430    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-14bca9d0-56877b5b86-ttsfz"]
Jul 09 19:18:17.892052 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:17.892016    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:18.085837 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:18.085793    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efaab627-e085-49d5-83ba-5d2fbda302ee" path="/var/lib/kubelet/pods/efaab627-e085-49d5-83ba-5d2fbda302ee/volumes"
Jul 09 19:18:22.638377 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:22.638343    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:22.707407 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:22.707375    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:22.859598 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:22.859557    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:22.966335 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:22.966302    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:24.043063 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.043017    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:24.122272 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.122240    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"]
Jul 09 19:18:24.122909 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.122884    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efaab627-e085-49d5-83ba-5d2fbda302ee" containerName="maas-api"
Jul 09 19:18:24.123088 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.123067    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="efaab627-e085-49d5-83ba-5d2fbda302ee" containerName="maas-api"
Jul 09 19:18:24.123291 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.123275    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="efaab627-e085-49d5-83ba-5d2fbda302ee" containerName="maas-api"
Jul 09 19:18:24.128376 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.128359    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.131015 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.130996    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-6ddac54e-serving-cert\""
Jul 09 19:18:24.140114 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.140088    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"]
Jul 09 19:18:24.143702 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.143675    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:24.226807 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.226772    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:24.239923 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.239894    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ef6e16-09fa-4170-82bb-a936807cc8ec-openshift-service-ca\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.240095 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.239942    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ef6e16-09fa-4170-82bb-a936807cc8ec-maas-api-tls\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.240095 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.240009    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-926vl\" (UniqueName: \"kubernetes.io/projected/d0ef6e16-09fa-4170-82bb-a936807cc8ec-kube-api-access-926vl\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.341306 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.341213    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-926vl\" (UniqueName: \"kubernetes.io/projected/d0ef6e16-09fa-4170-82bb-a936807cc8ec-kube-api-access-926vl\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.341490 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.341331    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ef6e16-09fa-4170-82bb-a936807cc8ec-openshift-service-ca\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.341490 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.341380    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ef6e16-09fa-4170-82bb-a936807cc8ec-maas-api-tls\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.341999 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.341978    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ef6e16-09fa-4170-82bb-a936807cc8ec-openshift-service-ca\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.344043 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.344012    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ef6e16-09fa-4170-82bb-a936807cc8ec-maas-api-tls\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.355790 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.355758    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-926vl\" (UniqueName: \"kubernetes.io/projected/d0ef6e16-09fa-4170-82bb-a936807cc8ec-kube-api-access-926vl\") pod \"maas-api-e2e-ait-6ddac54e-69d566c68-l55p2\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.373787 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.373758    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:24.441279 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.441245    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:24.584744 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.584720    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"]
Jul 09 19:18:24.586862 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:18:24.586830    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0ef6e16_09fa_4170_82bb_a936807cc8ec.slice/crio-140ba108683bdd8aa08124d409e50f516893ee0d7d5d934f3eedb6404fc742e4 WatchSource:0}: Error finding container 140ba108683bdd8aa08124d409e50f516893ee0d7d5d934f3eedb6404fc742e4: Status 404 returned error can't find the container with id 140ba108683bdd8aa08124d409e50f516893ee0d7d5d934f3eedb6404fc742e4
Jul 09 19:18:24.633421 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.633392    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:24.734023 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:24.733993    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:25.020862 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:25.020830    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2" event={"ID":"d0ef6e16-09fa-4170-82bb-a936807cc8ec","Type":"ContainerStarted","Data":"140ba108683bdd8aa08124d409e50f516893ee0d7d5d934f3eedb6404fc742e4"}
Jul 09 19:18:26.026959 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:26.026922    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2" event={"ID":"d0ef6e16-09fa-4170-82bb-a936807cc8ec","Type":"ContainerStarted","Data":"8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8"}
Jul 09 19:18:26.027363 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:26.027051    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:26.045772 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:26.045726    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2" podStartSLOduration=1.5618692360000002 podStartE2EDuration="2.045710172s" totalImagesPullingTime="483.840936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:18:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:18:26.044030054 +0000 UTC m=+1516.396465116" watchObservedRunningTime="2026-07-09 19:18:26.045710172 +0000 UTC m=+1516.398145239"
Jul 09 19:18:26.446104 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:26.446071    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:29.084144 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.084110    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:29.447132 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.447103    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"]
Jul 09 19:18:29.447332 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.447304    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2" podUID="d0ef6e16-09fa-4170-82bb-a936807cc8ec" containerName="maas-api" containerID="cri-o://8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8" gracePeriod=30
Jul 09 19:18:29.481698 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.481667    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:29.700499 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.700439    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:29.792615 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.792579    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ef6e16-09fa-4170-82bb-a936807cc8ec-maas-api-tls\") pod \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") "
Jul 09 19:18:29.792821 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.792669    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-926vl\" (UniqueName: \"kubernetes.io/projected/d0ef6e16-09fa-4170-82bb-a936807cc8ec-kube-api-access-926vl\") pod \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") "
Jul 09 19:18:29.792821 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.792708    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ef6e16-09fa-4170-82bb-a936807cc8ec-openshift-service-ca\") pod \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\" (UID: \"d0ef6e16-09fa-4170-82bb-a936807cc8ec\") "
Jul 09 19:18:29.793105 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.793081    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0ef6e16-09fa-4170-82bb-a936807cc8ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d0ef6e16-09fa-4170-82bb-a936807cc8ec" (UID: "d0ef6e16-09fa-4170-82bb-a936807cc8ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:18:29.794671 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.794648    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d0ef6e16-09fa-4170-82bb-a936807cc8ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d0ef6e16-09fa-4170-82bb-a936807cc8ec" (UID: "d0ef6e16-09fa-4170-82bb-a936807cc8ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:18:29.794800 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.794780    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0ef6e16-09fa-4170-82bb-a936807cc8ec-kube-api-access-926vl" (OuterVolumeSpecName: "kube-api-access-926vl") pod "d0ef6e16-09fa-4170-82bb-a936807cc8ec" (UID: "d0ef6e16-09fa-4170-82bb-a936807cc8ec"). InnerVolumeSpecName "kube-api-access-926vl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:18:29.893559 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.893494    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d0ef6e16-09fa-4170-82bb-a936807cc8ec-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:29.893559 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.893557    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-926vl\" (UniqueName: \"kubernetes.io/projected/d0ef6e16-09fa-4170-82bb-a936807cc8ec-kube-api-access-926vl\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:29.893766 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:29.893572    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d0ef6e16-09fa-4170-82bb-a936807cc8ec-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:30.049588 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.049468    2556 generic.go:356] "Generic (PLEG): container finished" podID="d0ef6e16-09fa-4170-82bb-a936807cc8ec" containerID="8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8" exitCode=0
Jul 09 19:18:30.049588 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.049513    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"
Jul 09 19:18:30.049588 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.049558    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2" event={"ID":"d0ef6e16-09fa-4170-82bb-a936807cc8ec","Type":"ContainerDied","Data":"8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8"}
Jul 09 19:18:30.049834 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.049622    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2" event={"ID":"d0ef6e16-09fa-4170-82bb-a936807cc8ec","Type":"ContainerDied","Data":"140ba108683bdd8aa08124d409e50f516893ee0d7d5d934f3eedb6404fc742e4"}
Jul 09 19:18:30.049834 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.049647    2556 scope.go:117] "RemoveContainer" containerID="8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8"
Jul 09 19:18:30.058709 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.058693    2556 scope.go:117] "RemoveContainer" containerID="8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8"
Jul 09 19:18:30.058981 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:18:30.058964    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8\": container with ID starting with 8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8 not found: ID does not exist" containerID="8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8"
Jul 09 19:18:30.059028 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.058992    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8"} err="failed to get container status \"8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8\": rpc error: code = NotFound desc = could not find container \"8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8\": container with ID starting with 8cfde4076f0136d3b67241273ddb4834a3a1872f1bf8b16983d579c84d7c34f8 not found: ID does not exist"
Jul 09 19:18:30.087018 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.086983    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"]
Jul 09 19:18:30.087018 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:30.087019    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-6ddac54e-69d566c68-l55p2"]
Jul 09 19:18:32.085399 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:32.085293    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0ef6e16-09fa-4170-82bb-a936807cc8ec" path="/var/lib/kubelet/pods/d0ef6e16-09fa-4170-82bb-a936807cc8ec/volumes"
Jul 09 19:18:35.209487 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:35.209453    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:35.309806 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:35.309766    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:37.641320 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:37.641289    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:41.455916 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:41.455883    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:41.559485 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:41.559448    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:41.710384 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:41.710291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:41.809971 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:41.809926    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:42.933257 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.933218    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:42.967681 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.967652    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"]
Jul 09 19:18:42.968062 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.968050    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0ef6e16-09fa-4170-82bb-a936807cc8ec" containerName="maas-api"
Jul 09 19:18:42.968102 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.968063    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0ef6e16-09fa-4170-82bb-a936807cc8ec" containerName="maas-api"
Jul 09 19:18:42.968164 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.968153    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0ef6e16-09fa-4170-82bb-a936807cc8ec" containerName="maas-api"
Jul 09 19:18:42.971249 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.971228    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:42.974054 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.974032    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2d937836-serving-cert\""
Jul 09 19:18:42.983684 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:42.983661    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"]
Jul 09 19:18:43.032588 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.032556    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:43.112820 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.112780    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.112820 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.112822    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7vndh\" (UniqueName: \"kubernetes.io/projected/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-kube-api-access-7vndh\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.113070 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.112940    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-openshift-service-ca\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.213995 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.213896    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.213995 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.213950    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7vndh\" (UniqueName: \"kubernetes.io/projected/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-kube-api-access-7vndh\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.214238 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.214030    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-openshift-service-ca\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.214238 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:18:43.214072    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-serving-cert: secret "maas-api-e2e-derive-2d937836-serving-cert" not found
Jul 09 19:18:43.214238 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:18:43.214171    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls podName:1575c91f-b2ca-45df-9ad5-6cf7fd33b7da nodeName:}" failed. No retries permitted until 2026-07-09 19:18:43.714147337 +0000 UTC m=+1534.066582406 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls") pod "maas-api-e2e-derive-2d937836-65d975998f-7t2t5" (UID: "1575c91f-b2ca-45df-9ad5-6cf7fd33b7da") : secret "maas-api-e2e-derive-2d937836-serving-cert" not found
Jul 09 19:18:43.214869 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.214846    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-openshift-service-ca\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.224658 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.224637    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7vndh\" (UniqueName: \"kubernetes.io/projected/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-kube-api-access-7vndh\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.435975 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.435942    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:43.534363 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.534274    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:43.717453 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.717420    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.719868 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.719847    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls\") pod \"maas-api-e2e-derive-2d937836-65d975998f-7t2t5\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:43.882646 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:43.882546    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:44.030638 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:44.030610    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"]
Jul 09 19:18:44.032888 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:18:44.032857    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1575c91f_b2ca_45df_9ad5_6cf7fd33b7da.slice/crio-f67e0d9a851abd6e9d62b460e65b43888323fd000b4d8041867ef3d95bcc3ca1 WatchSource:0}: Error finding container f67e0d9a851abd6e9d62b460e65b43888323fd000b4d8041867ef3d95bcc3ca1: Status 404 returned error can't find the container with id f67e0d9a851abd6e9d62b460e65b43888323fd000b4d8041867ef3d95bcc3ca1
Jul 09 19:18:44.127758 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:44.127717    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5" event={"ID":"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da","Type":"ContainerStarted","Data":"f67e0d9a851abd6e9d62b460e65b43888323fd000b4d8041867ef3d95bcc3ca1"}
Jul 09 19:18:45.135050 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:45.135013    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5" event={"ID":"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da","Type":"ContainerStarted","Data":"cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1"}
Jul 09 19:18:45.135443 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:45.135081    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:45.151981 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:45.151882    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5" podStartSLOduration=2.669216466 podStartE2EDuration="3.151859728s" totalImagesPullingTime="482.643262ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:18:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:18:45.151633067 +0000 UTC m=+1535.504068130" watchObservedRunningTime="2026-07-09 19:18:45.151859728 +0000 UTC m=+1535.504294797"
Jul 09 19:18:45.410996 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:45.410963    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:47.269378 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.269332    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:47.457319 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.457283    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:47.560712 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.560633    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:47.619519 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.619491    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"]
Jul 09 19:18:47.619724 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.619679    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5" podUID="1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" containerName="maas-api" containerID="cri-o://cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1" gracePeriod=30
Jul 09 19:18:47.651490 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.651465    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:47.874697 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.874671    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:47.956619 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.956589    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls\") pod \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") "
Jul 09 19:18:47.956796 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.956626    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7vndh\" (UniqueName: \"kubernetes.io/projected/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-kube-api-access-7vndh\") pod \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") "
Jul 09 19:18:47.956796 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.956658    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-openshift-service-ca\") pod \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\" (UID: \"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da\") "
Jul 09 19:18:47.957069 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.957036    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" (UID: "1575c91f-b2ca-45df-9ad5-6cf7fd33b7da"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:18:47.958755 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.958732    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-kube-api-access-7vndh" (OuterVolumeSpecName: "kube-api-access-7vndh") pod "1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" (UID: "1575c91f-b2ca-45df-9ad5-6cf7fd33b7da"). InnerVolumeSpecName "kube-api-access-7vndh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:18:47.958847 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:47.958747    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" (UID: "1575c91f-b2ca-45df-9ad5-6cf7fd33b7da"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:18:48.057972 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.057937    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:48.057972 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.057968    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7vndh\" (UniqueName: \"kubernetes.io/projected/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-kube-api-access-7vndh\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:48.057972 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.057980    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:18:48.152246 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.152150    2556 generic.go:356] "Generic (PLEG): container finished" podID="1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" containerID="cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1" exitCode=0
Jul 09 19:18:48.152246 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.152198    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"
Jul 09 19:18:48.152439 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.152239    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5" event={"ID":"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da","Type":"ContainerDied","Data":"cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1"}
Jul 09 19:18:48.152439 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.152291    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5" event={"ID":"1575c91f-b2ca-45df-9ad5-6cf7fd33b7da","Type":"ContainerDied","Data":"f67e0d9a851abd6e9d62b460e65b43888323fd000b4d8041867ef3d95bcc3ca1"}
Jul 09 19:18:48.152439 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.152312    2556 scope.go:117] "RemoveContainer" containerID="cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1"
Jul 09 19:18:48.161341 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.161324    2556 scope.go:117] "RemoveContainer" containerID="cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1"
Jul 09 19:18:48.161599 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:18:48.161579    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1\": container with ID starting with cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1 not found: ID does not exist" containerID="cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1"
Jul 09 19:18:48.161653 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.161611    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1"} err="failed to get container status \"cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1\": rpc error: code = NotFound desc = could not find container \"cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1\": container with ID starting with cb00222ce5ca97fc788fc94fe7bb7344dd7a370123e4d6715f4b4590fff9e9f1 not found: ID does not exist"
Jul 09 19:18:48.169360 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.169336    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"]
Jul 09 19:18:48.177988 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:48.177964    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2d937836-65d975998f-7t2t5"]
Jul 09 19:18:50.034216 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:50.034181    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:50.085623 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:50.085588    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" path="/var/lib/kubelet/pods/1575c91f-b2ca-45df-9ad5-6cf7fd33b7da/volumes"
Jul 09 19:18:53.433495 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:53.433447    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:53.533988 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:53.533957    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:53.943276 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:53.943245    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:54.030859 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:54.030823    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:54.130930 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:54.130895    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:18:55.526089 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:18:55.526048    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:01.718703 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:01.718668    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:01.789804 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:01.789768    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:01.912093 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:01.912059    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:02.008555 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:02.008444    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:02.472921 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:02.472891    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:18.849873 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:18.849842    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:28.450990 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:28.450956    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:34.955775 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:34.955740    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:35.033151 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:35.033120    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:35.118999 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:35.118962    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:35.323043 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:35.322948    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:51.098885 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:51.098850    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:57.147583 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:57.147519    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:57.259760 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:57.259732    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:57.349576 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:57.349513    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:19:59.171132 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:19:59.171093    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:07.897053 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:07.896963    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:08.715608 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:08.715574    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:14.796180 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:14.796149    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:14.898365 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:14.898334    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:15.000592 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:15.000559    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:15.100471 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:15.100386    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:15.876840 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:15.876804    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:15.983054 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:15.983019    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:16.080714 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:16.080681    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:17.738264 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:17.738221    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:23.966839 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:23.966808    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:24.067659 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:24.067622    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:24.622413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:24.622378    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:30.967437 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:30.967395    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:31.067779 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:31.067741    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:31.222430 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:31.222331    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:31.554030 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:31.553945    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:32.645504 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:32.645466    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:34.259348 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.259318    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:34.327974 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.327940    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"]
Jul 09 19:20:34.328366 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.328354    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" containerName="maas-api"
Jul 09 19:20:34.328415 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.328368    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" containerName="maas-api"
Jul 09 19:20:34.328450 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.328431    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1575c91f-b2ca-45df-9ad5-6cf7fd33b7da" containerName="maas-api"
Jul 09 19:20:34.331795 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.331778    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.334403 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.334386    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-87c27070-serving-cert\""
Jul 09 19:20:34.342147 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.342121    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"]
Jul 09 19:20:34.343250 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.343224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:34.371745 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.371714    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7468\" (UniqueName: \"kubernetes.io/projected/5a3ee43d-fd53-4c01-8253-6fe24095a698-kube-api-access-n7468\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.371918 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.371778    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.371918 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.371843    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3ee43d-fd53-4c01-8253-6fe24095a698-openshift-service-ca\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.472728 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.472687    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.472946 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.472740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3ee43d-fd53-4c01-8253-6fe24095a698-openshift-service-ca\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.472946 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:20:34.472853    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-serving-cert: secret "maas-api-e2e-mt-87c27070-serving-cert" not found
Jul 09 19:20:34.472946 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:20:34.472922    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls podName:5a3ee43d-fd53-4c01-8253-6fe24095a698 nodeName:}" failed. No retries permitted until 2026-07-09 19:20:34.972904692 +0000 UTC m=+1645.325339748 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls") pod "maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr" (UID: "5a3ee43d-fd53-4c01-8253-6fe24095a698") : secret "maas-api-e2e-mt-87c27070-serving-cert" not found
Jul 09 19:20:34.472946 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.472939    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7468\" (UniqueName: \"kubernetes.io/projected/5a3ee43d-fd53-4c01-8253-6fe24095a698-kube-api-access-n7468\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.473572 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.473512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3ee43d-fd53-4c01-8253-6fe24095a698-openshift-service-ca\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.482050 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.482024    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7468\" (UniqueName: \"kubernetes.io/projected/5a3ee43d-fd53-4c01-8253-6fe24095a698-kube-api-access-n7468\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.534775 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.534681    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:34.635825 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.635791    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:34.977596 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.977563    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:34.980052 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:34.980021    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls\") pod \"maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:35.006393 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:35.006355    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:35.108251 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:35.108223    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:35.243015 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:35.242930    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:35.393129 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:35.393106    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"]
Jul 09 19:20:35.395810 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:20:35.395779    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a3ee43d_fd53_4c01_8253_6fe24095a698.slice/crio-ef42089fccb60a8e9cf12bbc0a1dffa09c4632ed29c5ba521445562d3cb5d691 WatchSource:0}: Error finding container ef42089fccb60a8e9cf12bbc0a1dffa09c4632ed29c5ba521445562d3cb5d691: Status 404 returned error can't find the container with id ef42089fccb60a8e9cf12bbc0a1dffa09c4632ed29c5ba521445562d3cb5d691
Jul 09 19:20:35.397333 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:35.397317    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:20:35.708814 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:35.708780    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr" event={"ID":"5a3ee43d-fd53-4c01-8253-6fe24095a698","Type":"ContainerStarted","Data":"ef42089fccb60a8e9cf12bbc0a1dffa09c4632ed29c5ba521445562d3cb5d691"}
Jul 09 19:20:36.715480 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:36.715446    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr" event={"ID":"5a3ee43d-fd53-4c01-8253-6fe24095a698","Type":"ContainerStarted","Data":"3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383"}
Jul 09 19:20:36.715894 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:36.715580    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:36.732453 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:36.732409    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr" podStartSLOduration=2.178828484 podStartE2EDuration="2.732392504s" totalImagesPullingTime="553.56402ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:20:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:20:36.730781049 +0000 UTC m=+1647.083216116" watchObservedRunningTime="2026-07-09 19:20:36.732392504 +0000 UTC m=+1647.084827569"
Jul 09 19:20:38.750488 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:38.750447    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:39.425399 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:39.425365    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:39.523214 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:39.523175    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:39.872244 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:39.872149    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:42.725990 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:42.725962    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:46.010324 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:46.010286    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:46.076138 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:46.076106    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:46.433170 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:46.433134    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:46.921862 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:46.921835    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:46.969710 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:46.969678    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"]
Jul 09 19:20:46.969920 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:46.969894    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr" podUID="5a3ee43d-fd53-4c01-8253-6fe24095a698" containerName="maas-api" containerID="cri-o://3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383" gracePeriod=30
Jul 09 19:20:47.022189 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.022159    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:47.230250 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.230224    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:47.296872 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.296840    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7468\" (UniqueName: \"kubernetes.io/projected/5a3ee43d-fd53-4c01-8253-6fe24095a698-kube-api-access-n7468\") pod \"5a3ee43d-fd53-4c01-8253-6fe24095a698\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") "
Jul 09 19:20:47.297062 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.296958    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3ee43d-fd53-4c01-8253-6fe24095a698-openshift-service-ca\") pod \"5a3ee43d-fd53-4c01-8253-6fe24095a698\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") "
Jul 09 19:20:47.297062 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.297004    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls\") pod \"5a3ee43d-fd53-4c01-8253-6fe24095a698\" (UID: \"5a3ee43d-fd53-4c01-8253-6fe24095a698\") "
Jul 09 19:20:47.297358 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.297328    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a3ee43d-fd53-4c01-8253-6fe24095a698-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5a3ee43d-fd53-4c01-8253-6fe24095a698" (UID: "5a3ee43d-fd53-4c01-8253-6fe24095a698"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:20:47.298944 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.298915    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a3ee43d-fd53-4c01-8253-6fe24095a698-kube-api-access-n7468" (OuterVolumeSpecName: "kube-api-access-n7468") pod "5a3ee43d-fd53-4c01-8253-6fe24095a698" (UID: "5a3ee43d-fd53-4c01-8253-6fe24095a698"). InnerVolumeSpecName "kube-api-access-n7468". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:20:47.299039 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.299017    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5a3ee43d-fd53-4c01-8253-6fe24095a698" (UID: "5a3ee43d-fd53-4c01-8253-6fe24095a698"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:20:47.398112 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.398077    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5a3ee43d-fd53-4c01-8253-6fe24095a698-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:20:47.398112 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.398105    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5a3ee43d-fd53-4c01-8253-6fe24095a698-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:20:47.398112 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.398114    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7468\" (UniqueName: \"kubernetes.io/projected/5a3ee43d-fd53-4c01-8253-6fe24095a698-kube-api-access-n7468\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:20:47.771099 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.771067    2556 generic.go:356] "Generic (PLEG): container finished" podID="5a3ee43d-fd53-4c01-8253-6fe24095a698" containerID="3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383" exitCode=0
Jul 09 19:20:47.771275 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.771115    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"
Jul 09 19:20:47.771275 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.771153    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr" event={"ID":"5a3ee43d-fd53-4c01-8253-6fe24095a698","Type":"ContainerDied","Data":"3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383"}
Jul 09 19:20:47.771275 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.771206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr" event={"ID":"5a3ee43d-fd53-4c01-8253-6fe24095a698","Type":"ContainerDied","Data":"ef42089fccb60a8e9cf12bbc0a1dffa09c4632ed29c5ba521445562d3cb5d691"}
Jul 09 19:20:47.771275 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.771235    2556 scope.go:117] "RemoveContainer" containerID="3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383"
Jul 09 19:20:47.785304 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.785268    2556 scope.go:117] "RemoveContainer" containerID="3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383"
Jul 09 19:20:47.785679 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:20:47.785644    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383\": container with ID starting with 3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383 not found: ID does not exist" containerID="3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383"
Jul 09 19:20:47.785797 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.785687    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383"} err="failed to get container status \"3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383\": rpc error: code = NotFound desc = could not find container \"3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383\": container with ID starting with 3ac62dd33384a6391528ad0f2fd49e64a80a88e6d00f8ad2e3f651e47e104383 not found: ID does not exist"
Jul 09 19:20:47.802813 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.802783    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"]
Jul 09 19:20:47.806787 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:47.806762    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-87c27070-665f4cdb59-fdgzr"]
Jul 09 19:20:48.084879 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:48.084795    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5a3ee43d-fd53-4c01-8253-6fe24095a698" path="/var/lib/kubelet/pods/5a3ee43d-fd53-4c01-8253-6fe24095a698/volumes"
Jul 09 19:20:53.149915 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:53.149885    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:53.219811 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:53.219776    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:53.425881 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:53.425850    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:53.659549 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:53.656891    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:53.781349 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:53.781224    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:53.876334 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:53.876293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:20:53.998093 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:20:53.998061    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:00.221209 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:00.221161    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:00.376040 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:00.376004    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:00.551173 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:00.551069    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:00.648782 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:00.648749    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:06.506747 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:06.506713    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:06.604616 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:06.604585    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:13.327632 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:13.327598    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:13.652270 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:13.652190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:14.062997 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:14.062963    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:16.241496 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.241457    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:16.310744 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.310712    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"]
Jul 09 19:21:16.311229 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.311199    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5a3ee43d-fd53-4c01-8253-6fe24095a698" containerName="maas-api"
Jul 09 19:21:16.311229 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.311228    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5a3ee43d-fd53-4c01-8253-6fe24095a698" containerName="maas-api"
Jul 09 19:21:16.311398 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.311325    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5a3ee43d-fd53-4c01-8253-6fe24095a698" containerName="maas-api"
Jul 09 19:21:16.315268 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.315249    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.318143 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.318120    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-bb59c5dc-serving-cert\""
Jul 09 19:21:16.329673 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.329649    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"]
Jul 09 19:21:16.349073 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.349047    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:16.373795 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.373767    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9335b922-15f5-4d26-b48a-1bcf95901e84-openshift-service-ca\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.373970 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.373812    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tlfp9\" (UniqueName: \"kubernetes.io/projected/9335b922-15f5-4d26-b48a-1bcf95901e84-kube-api-access-tlfp9\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.373970 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.373877    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.443125 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.443090    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:16.475333 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.475301    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9335b922-15f5-4d26-b48a-1bcf95901e84-openshift-service-ca\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.475513 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.475345    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tlfp9\" (UniqueName: \"kubernetes.io/projected/9335b922-15f5-4d26-b48a-1bcf95901e84-kube-api-access-tlfp9\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.475513 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.475387    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.475513 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:21:16.475500    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-serving-cert: secret "maas-api-e2e-mt-bb59c5dc-serving-cert" not found
Jul 09 19:21:16.475681 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:21:16.475587    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls podName:9335b922-15f5-4d26-b48a-1bcf95901e84 nodeName:}" failed. No retries permitted until 2026-07-09 19:21:16.97556793 +0000 UTC m=+1687.328002973 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls") pod "maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb" (UID: "9335b922-15f5-4d26-b48a-1bcf95901e84") : secret "maas-api-e2e-mt-bb59c5dc-serving-cert" not found
Jul 09 19:21:16.476016 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.475996    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9335b922-15f5-4d26-b48a-1bcf95901e84-openshift-service-ca\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.494352 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.494287    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tlfp9\" (UniqueName: \"kubernetes.io/projected/9335b922-15f5-4d26-b48a-1bcf95901e84-kube-api-access-tlfp9\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.704474 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.704440    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:16.845277 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.845198    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:16.945934 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.945887    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:16.980168 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.980138    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:16.982597 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:16.982571    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls\") pod \"maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:17.226160 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:17.226124    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:17.374488 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:17.374458    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"]
Jul 09 19:21:17.375801 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:21:17.375773    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9335b922_15f5_4d26_b48a_1bcf95901e84.slice/crio-635b85fbb3623ebd6e4fb04f4796ff57d17143701b837480ad6c607e58d4afb4 WatchSource:0}: Error finding container 635b85fbb3623ebd6e4fb04f4796ff57d17143701b837480ad6c607e58d4afb4: Status 404 returned error can't find the container with id 635b85fbb3623ebd6e4fb04f4796ff57d17143701b837480ad6c607e58d4afb4
Jul 09 19:21:17.930993 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:17.930955    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb" event={"ID":"9335b922-15f5-4d26-b48a-1bcf95901e84","Type":"ContainerStarted","Data":"19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332"}
Jul 09 19:21:17.931171 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:17.931035    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb" event={"ID":"9335b922-15f5-4d26-b48a-1bcf95901e84","Type":"ContainerStarted","Data":"635b85fbb3623ebd6e4fb04f4796ff57d17143701b837480ad6c607e58d4afb4"}
Jul 09 19:21:17.931171 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:17.931060    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:17.967002 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:17.966952    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb" podStartSLOduration=1.588830054 podStartE2EDuration="1.966936534s" totalImagesPullingTime="378.10648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:21:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:21:17.963386833 +0000 UTC m=+1688.315821897" watchObservedRunningTime="2026-07-09 19:21:17.966936534 +0000 UTC m=+1688.319371737"
Jul 09 19:21:19.119381 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:19.119352    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:19.208390 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:19.208357    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:20.368660 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:20.368623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:20.468431 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:20.468400    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:20.727243 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:20.727209    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:23.941201 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:23.941172    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:37.721268 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:37.721180    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:37.821249 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:37.821218    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:37.890796 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:37.890761    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:58.275349 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.275289    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:58.658357 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.658279    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"]
Jul 09 19:21:58.658570 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.658542    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb" podUID="9335b922-15f5-4d26-b48a-1bcf95901e84" containerName="maas-api" containerID="cri-o://19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332" gracePeriod=30
Jul 09 19:21:58.681063 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.681037    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:21:58.916325 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.916267    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:58.958681 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.958647    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9335b922-15f5-4d26-b48a-1bcf95901e84-openshift-service-ca\") pod \"9335b922-15f5-4d26-b48a-1bcf95901e84\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") "
Jul 09 19:21:58.958858 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.958710    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tlfp9\" (UniqueName: \"kubernetes.io/projected/9335b922-15f5-4d26-b48a-1bcf95901e84-kube-api-access-tlfp9\") pod \"9335b922-15f5-4d26-b48a-1bcf95901e84\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") "
Jul 09 19:21:58.958858 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.958727    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls\") pod \"9335b922-15f5-4d26-b48a-1bcf95901e84\" (UID: \"9335b922-15f5-4d26-b48a-1bcf95901e84\") "
Jul 09 19:21:58.959084 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.959053    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9335b922-15f5-4d26-b48a-1bcf95901e84-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9335b922-15f5-4d26-b48a-1bcf95901e84" (UID: "9335b922-15f5-4d26-b48a-1bcf95901e84"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:21:58.960881 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.960856    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9335b922-15f5-4d26-b48a-1bcf95901e84-kube-api-access-tlfp9" (OuterVolumeSpecName: "kube-api-access-tlfp9") pod "9335b922-15f5-4d26-b48a-1bcf95901e84" (UID: "9335b922-15f5-4d26-b48a-1bcf95901e84"). InnerVolumeSpecName "kube-api-access-tlfp9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:21:58.961008 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:58.960953    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9335b922-15f5-4d26-b48a-1bcf95901e84" (UID: "9335b922-15f5-4d26-b48a-1bcf95901e84"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:21:59.059751 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.059694    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9335b922-15f5-4d26-b48a-1bcf95901e84-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:21:59.059751 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.059749    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tlfp9\" (UniqueName: \"kubernetes.io/projected/9335b922-15f5-4d26-b48a-1bcf95901e84-kube-api-access-tlfp9\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:21:59.059751 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.059759    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9335b922-15f5-4d26-b48a-1bcf95901e84-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:21:59.147537 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.147506    2556 generic.go:356] "Generic (PLEG): container finished" podID="9335b922-15f5-4d26-b48a-1bcf95901e84" containerID="19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332" exitCode=0
Jul 09 19:21:59.147730 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.147581    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"
Jul 09 19:21:59.147730 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.147599    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb" event={"ID":"9335b922-15f5-4d26-b48a-1bcf95901e84","Type":"ContainerDied","Data":"19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332"}
Jul 09 19:21:59.147730 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.147662    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb" event={"ID":"9335b922-15f5-4d26-b48a-1bcf95901e84","Type":"ContainerDied","Data":"635b85fbb3623ebd6e4fb04f4796ff57d17143701b837480ad6c607e58d4afb4"}
Jul 09 19:21:59.147730 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.147693    2556 scope.go:117] "RemoveContainer" containerID="19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332"
Jul 09 19:21:59.157195 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.157177    2556 scope.go:117] "RemoveContainer" containerID="19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332"
Jul 09 19:21:59.157442 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:21:59.157424    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332\": container with ID starting with 19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332 not found: ID does not exist" containerID="19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332"
Jul 09 19:21:59.157496 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.157450    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332"} err="failed to get container status \"19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332\": rpc error: code = NotFound desc = could not find container \"19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332\": container with ID starting with 19d48b51997e39f9874a8356dca011e4f5903bf89fc9c4e9658f8dafa26ff332 not found: ID does not exist"
Jul 09 19:21:59.169932 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.169910    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"]
Jul 09 19:21:59.175715 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:21:59.175692    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-bb59c5dc-5f7cd64f74-g4qpb"]
Jul 09 19:22:00.085536 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:00.085490    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9335b922-15f5-4d26-b48a-1bcf95901e84" path="/var/lib/kubelet/pods/9335b922-15f5-4d26-b48a-1bcf95901e84/volumes"
Jul 09 19:22:04.353933 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:04.353867    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:09.435730 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:09.435678    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:45.628032 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:45.628001    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:47.066358 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:47.066322    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:52.486948 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:52.486912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:52.783284 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:52.783193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:52.947441 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:52.947400    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:53.042097 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:53.042009    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:54.175106 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:54.175071    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:54.274122 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:54.274088    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:54.373925 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:54.373891    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:55.113737 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:55.113708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:55.211817 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:55.211784    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:55.312400 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:55.312361    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:22:56.685708 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:22:56.685661    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:02.937367 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:02.937279    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:03.037693 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:03.037658    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:03.623719 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:03.623652    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:09.636792 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:09.636755    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:09.793869 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:09.793829    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:09.892133 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:09.892036    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:10.167860 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:10.167818    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:23:10.172026 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:10.171998    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:23:10.174005 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:10.173984    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:23:10.178251 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:10.178236    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:23:20.370962 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:20.370930    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:20.714717 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:20.714676    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:41.691578 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:41.691537    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:47.769387 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:47.769349    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:47.838585 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:47.838557    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:48.147916 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:48.147838    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:48.460075 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:48.460023    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:48.628992 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:48.628948    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:50.946422 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:50.946386    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:51.019626 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.019589    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"]
Jul 09 19:23:51.020334 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.020311    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9335b922-15f5-4d26-b48a-1bcf95901e84" containerName="maas-api"
Jul 09 19:23:51.020334 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.020332    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9335b922-15f5-4d26-b48a-1bcf95901e84" containerName="maas-api"
Jul 09 19:23:51.020505 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.020485    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9335b922-15f5-4d26-b48a-1bcf95901e84" containerName="maas-api"
Jul 09 19:23:51.024137 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.024116    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.026633 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.026614    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c6e06b-serving-cert\""
Jul 09 19:23:51.036167 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.036145    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"]
Jul 09 19:23:51.036803 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.036781    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:51.075206 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.075174    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cntg\" (UniqueName: \"kubernetes.io/projected/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-kube-api-access-6cntg\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.075379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.075241    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.075379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.075303    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.111348 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.111299    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:51.176876 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.176845    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.177052 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.176909    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cntg\" (UniqueName: \"kubernetes.io/projected/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-kube-api-access-6cntg\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.177052 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.176948    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.177141 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:23:51.177069    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-serving-cert: secret "maas-api-e2e-inf-a-c6e06b-serving-cert" not found
Jul 09 19:23:51.177141 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:23:51.177124    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls podName:3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb nodeName:}" failed. No retries permitted until 2026-07-09 19:23:51.677107574 +0000 UTC m=+1842.029542618 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls") pod "maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw" (UID: "3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb") : secret "maas-api-e2e-inf-a-c6e06b-serving-cert" not found
Jul 09 19:23:51.177661 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.177639    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.186326 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.186299    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cntg\" (UniqueName: \"kubernetes.io/projected/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-kube-api-access-6cntg\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.378457 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.378359    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:51.535493 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.535454    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:51.636156 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.636071    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:51.682094 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.682060    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.684497 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.684474    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls\") pod \"maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:51.937920 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:51.937884    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:52.086899 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:23:52.086866    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f5dbc4a_1d74_4f45_ac26_6ab3128ae6fb.slice/crio-e4fbc8003144482adbae22524d51e125c181f8a9e59abec7b998a09b6a900cc7 WatchSource:0}: Error finding container e4fbc8003144482adbae22524d51e125c181f8a9e59abec7b998a09b6a900cc7: Status 404 returned error can't find the container with id e4fbc8003144482adbae22524d51e125c181f8a9e59abec7b998a09b6a900cc7
Jul 09 19:23:52.088110 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:52.088086    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"]
Jul 09 19:23:52.738723 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:52.738680    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw" event={"ID":"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb","Type":"ContainerStarted","Data":"e4fbc8003144482adbae22524d51e125c181f8a9e59abec7b998a09b6a900cc7"}
Jul 09 19:23:53.745737 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:53.745702    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw" event={"ID":"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb","Type":"ContainerStarted","Data":"64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5"}
Jul 09 19:23:53.746213 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:53.745890    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:53.764304 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:53.764249    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw" podStartSLOduration=3.009898548 podStartE2EDuration="3.764232333s" totalImagesPullingTime="754.333785ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:23:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:23:53.763445357 +0000 UTC m=+1844.115880423" watchObservedRunningTime="2026-07-09 19:23:53.764232333 +0000 UTC m=+1844.116667398"
Jul 09 19:23:55.462050 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:55.462018    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:55.813038 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:55.812948    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:56.112012 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.111926    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"]
Jul 09 19:23:56.115750 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.115731    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.118331 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.118310    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 19:23:56.118447 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.118383    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-eb3d8a-openshift-default-dockercfg-vvqfr\""
Jul 09 19:23:56.128023 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.127998    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"]
Jul 09 19:23:56.171045 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.171008    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:56.223251 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223215    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-envoy\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223426 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223265    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-socket\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223426 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223293    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc2e621-86c7-4efa-801d-b31894185a08-istio-podinfo\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223426 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223316    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-istio-token\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223426 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223346    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-credential-socket\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223426 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223410    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lqfrf\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-kube-api-access-lqfrf\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223678 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223440    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2e621-86c7-4efa-801d-b31894185a08-istiod-ca-cert\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223678 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223464    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-certs\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.223678 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.223486    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-data\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.306628 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.306590    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:23:56.324073 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324048    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-socket\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324222 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324080    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc2e621-86c7-4efa-801d-b31894185a08-istio-podinfo\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324222 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324097    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-istio-token\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324222 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324125    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-credential-socket\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324222 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324151    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lqfrf\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-kube-api-access-lqfrf\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324222 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324176    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2e621-86c7-4efa-801d-b31894185a08-istiod-ca-cert\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324222 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-certs\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324230    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-data\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324288    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-envoy\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-socket\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324740 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324573    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-credential-socket\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324740 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324711    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-data\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.324819 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.324754    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-certs\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.325546 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.325493    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2e621-86c7-4efa-801d-b31894185a08-istiod-ca-cert\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.327090 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.327071    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-envoy\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.327240 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.327223    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc2e621-86c7-4efa-801d-b31894185a08-istio-podinfo\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.332623 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.332603    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-istio-token\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.332782 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.332761    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lqfrf\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-kube-api-access-lqfrf\") pod \"e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") " pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.428245 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.428221    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:23:56.578062 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.578028    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"]
Jul 09 19:23:56.579786 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:23:56.579753    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dc2e621_86c7_4efa_801d_b31894185a08.slice/crio-7953e49d220caaa23ca1e3a7722789e48a49aea76486c5141d46f5d3985b4ea9 WatchSource:0}: Error finding container 7953e49d220caaa23ca1e3a7722789e48a49aea76486c5141d46f5d3985b4ea9: Status 404 returned error can't find the container with id 7953e49d220caaa23ca1e3a7722789e48a49aea76486c5141d46f5d3985b4ea9
Jul 09 19:23:56.761564 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:56.761448    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" event={"ID":"5dc2e621-86c7-4efa-801d-b31894185a08","Type":"ContainerStarted","Data":"7953e49d220caaa23ca1e3a7722789e48a49aea76486c5141d46f5d3985b4ea9"}
Jul 09 19:23:59.466022 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:59.465987    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:23:59.466292 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:59.466061    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:23:59.466292 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:59.466091    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:23:59.756224 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:59.756147    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:23:59.782463 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:59.782203    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" event={"ID":"5dc2e621-86c7-4efa-801d-b31894185a08","Type":"ContainerStarted","Data":"471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3"}
Jul 09 19:23:59.803419 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:23:59.803357    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" podStartSLOduration=0.919688913 podStartE2EDuration="3.803334845s" totalImagesPullingTime="2.883645932s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:23:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:23:59.800369281 +0000 UTC m=+1850.152804349" watchObservedRunningTime="2026-07-09 19:23:59.803334845 +0000 UTC m=+1850.155769917"
Jul 09 19:24:00.428926 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:00.428893    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:00.433984 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:00.433959    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:00.793373 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:00.793240    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:00.794561 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:00.794517    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:01.703102 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:01.703060    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:01.802885 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:01.802851    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:01.970318 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:01.970193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:02.346294 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:02.346170    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:02.446391 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:02.446354    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:02.548352 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:02.548293    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:06.209401 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:06.209366    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:13.143519 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:13.143472    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:13.244128 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:13.244089    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:14.864443 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:14.864410    2556 generic.go:356] "Generic (PLEG): container finished" podID="5dc2e621-86c7-4efa-801d-b31894185a08" containerID="471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3" exitCode=0
Jul 09 19:24:14.864867 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:14.864484    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" event={"ID":"5dc2e621-86c7-4efa-801d-b31894185a08","Type":"ContainerDied","Data":"471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3"}
Jul 09 19:24:14.864910 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:14.864902    2556 scope.go:117] "RemoveContainer" containerID="471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3"
Jul 09 19:24:14.866985 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:14.866946    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:24:14.867121 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:14.867025    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:24:14.867121 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:14.867055    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:24:15.793496 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:15.793454    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:15.873452 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:15.873411    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" event={"ID":"5dc2e621-86c7-4efa-801d-b31894185a08","Type":"ContainerStarted","Data":"3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7"}
Jul 09 19:24:16.428650 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:16.428609    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:16.429609 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:16.429586    2556 patch_prober.go:28] interesting pod/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 19:24:16.429671 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:16.429634    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 19:24:17.429199 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:17.429163    2556 patch_prober.go:28] interesting pod/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 19:24:17.429599 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:17.429217    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 19:24:18.429485 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:18.429444    2556 patch_prober.go:28] interesting pod/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 19:24:18.429959 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:18.429516    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 19:24:19.429225 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:19.429193    2556 patch_prober.go:28] interesting pod/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused" start-of-body=
Jul 09 19:24:19.429394 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:19.429248    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.61:15021/healthz/ready\": dial tcp 10.132.0.61:15021: connect: connection refused"
Jul 09 19:24:20.432595 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:20.432567    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:20.433081 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:20.432946    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:20.433580 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:20.433561    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:24:24.138566 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:24.138504    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:24.239567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:24.239521    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:30.941021 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:30.940990    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:24:49.993704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:24:49.993675    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:07.734905 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:07.734853    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:13.042810 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:13.042750    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:30.453741 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:30.453707    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:31.340899 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:31.340858    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:31.537150 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:31.537119    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:31.982084 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:31.982051    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:32.120776 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.120738    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:32.220340 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.220301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:32.349492 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.348888    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"]
Jul 09 19:25:32.349492 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.349232    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw" podUID="3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" containerName="maas-api" containerID="cri-o://64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5" gracePeriod=30
Jul 09 19:25:32.383886 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.383857    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:32.611993 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.611928    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:25:32.712415 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.712383    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls\") pod \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") "
Jul 09 19:25:32.712626 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.712447    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-openshift-service-ca\") pod \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") "
Jul 09 19:25:32.712626 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.712517    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6cntg\" (UniqueName: \"kubernetes.io/projected/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-kube-api-access-6cntg\") pod \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\" (UID: \"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb\") "
Jul 09 19:25:32.712838 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.712798    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" (UID: "3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:25:32.714610 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.714578    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" (UID: "3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:25:32.714748 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.714605    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-kube-api-access-6cntg" (OuterVolumeSpecName: "kube-api-access-6cntg") pod "3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" (UID: "3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb"). InnerVolumeSpecName "kube-api-access-6cntg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:25:32.813304 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.813263    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:25:32.813304 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.813297    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6cntg\" (UniqueName: \"kubernetes.io/projected/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-kube-api-access-6cntg\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:25:32.813304 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:32.813308    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:25:33.304692 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.304664    2556 generic.go:356] "Generic (PLEG): container finished" podID="3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" containerID="64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5" exitCode=0
Jul 09 19:25:33.304861 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.304704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw" event={"ID":"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb","Type":"ContainerDied","Data":"64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5"}
Jul 09 19:25:33.304861 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.304711    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"
Jul 09 19:25:33.304861 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.304741    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw" event={"ID":"3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb","Type":"ContainerDied","Data":"e4fbc8003144482adbae22524d51e125c181f8a9e59abec7b998a09b6a900cc7"}
Jul 09 19:25:33.304861 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.304763    2556 scope.go:117] "RemoveContainer" containerID="64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5"
Jul 09 19:25:33.314416 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.314396    2556 scope.go:117] "RemoveContainer" containerID="64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5"
Jul 09 19:25:33.314706 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:25:33.314684    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5\": container with ID starting with 64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5 not found: ID does not exist" containerID="64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5"
Jul 09 19:25:33.314764 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.314724    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5"} err="failed to get container status \"64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5\": rpc error: code = NotFound desc = could not find container \"64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5\": container with ID starting with 64c667de95574d1721aa08c6eea62ea88ebda3858a51c52d726fc7efaf48cfc5 not found: ID does not exist"
Jul 09 19:25:33.326506 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.326481    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"]
Jul 09 19:25:33.331472 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:33.331449    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c6e06b-6854774fd9-dbgqw"]
Jul 09 19:25:34.085452 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:34.085418    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" path="/var/lib/kubelet/pods/3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb/volumes"
Jul 09 19:25:37.183028 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:37.182995    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:57.694739 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:57.694701    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:25:57.874747 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:25:57.874712    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:14.128661 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:14.128623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:14.401011 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:14.400905    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:14.646011 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:14.645981    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:14.802410 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:14.802374    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:15.224819 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:15.224789    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:15.454805 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:15.454770    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:15.556112 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:15.556017    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:15.717788 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:15.717745    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:21.201681 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.201645    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:21.303173 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.303113    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:21.403050 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.403016    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:21.503745 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.503674    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:21.633117 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.633082    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"]
Jul 09 19:26:21.633390 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.633335    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy" containerID="cri-o://3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7" gracePeriod=30
Jul 09 19:26:21.655749 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.655719    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:21.880336 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:21.880245    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:22.232331 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:22.232301    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:22.554871 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:22.554786    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:22.653912 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:22.653882    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:25.643010 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:25.642976    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:25.735309 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:25.735272    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:25.802404 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:25.802367    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:26.324390 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:26.324360    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:26.426768 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:26.426736    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:26.525715 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:26.525671    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:26.899631 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:26.899351    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:26:27.019481 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019440    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-istio-token\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.019704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019514    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-data\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.019704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019570    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2e621-86c7-4efa-801d-b31894185a08-istiod-ca-cert\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.019704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019595    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-socket\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.019704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019613    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc2e621-86c7-4efa-801d-b31894185a08-istio-podinfo\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.019704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019636    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-credential-socket\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.019704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019669    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-certs\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.020018 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019719    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lqfrf\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-kube-api-access-lqfrf\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.020018 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.019749    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-envoy\") pod \"5dc2e621-86c7-4efa-801d-b31894185a08\" (UID: \"5dc2e621-86c7-4efa-801d-b31894185a08\") "
Jul 09 19:26:27.020135 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.020049    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:26:27.020135 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.020088    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:26:27.020258 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.020144    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dc2e621-86c7-4efa-801d-b31894185a08-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:26:27.020258 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.020165    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:26:27.020620 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.020593    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-data" (OuterVolumeSpecName: "istio-data") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:26:27.021813 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.021781    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-istio-token" (OuterVolumeSpecName: "istio-token") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:26:27.022065 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.022043    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-kube-api-access-lqfrf" (OuterVolumeSpecName: "kube-api-access-lqfrf") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "kube-api-access-lqfrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:26:27.022125 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.022046    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5dc2e621-86c7-4efa-801d-b31894185a08-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 19:26:27.022125 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.022045    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5dc2e621-86c7-4efa-801d-b31894185a08" (UID: "5dc2e621-86c7-4efa-801d-b31894185a08"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:26:27.121442 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121406    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-data\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121442 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121435    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5dc2e621-86c7-4efa-801d-b31894185a08-istiod-ca-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121442 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121444    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-socket\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121442 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121452    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5dc2e621-86c7-4efa-801d-b31894185a08-istio-podinfo\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121828 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121460    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-credential-socket\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121828 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121469    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-workload-certs\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121828 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121477    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lqfrf\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-kube-api-access-lqfrf\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121828 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121484    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5dc2e621-86c7-4efa-801d-b31894185a08-istio-envoy\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.121828 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.121492    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5dc2e621-86c7-4efa-801d-b31894185a08-istio-token\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:27.603065 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.603034    2556 generic.go:356] "Generic (PLEG): container finished" podID="5dc2e621-86c7-4efa-801d-b31894185a08" containerID="3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7" exitCode=0
Jul 09 19:26:27.603248 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.603093    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"
Jul 09 19:26:27.603248 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.603115    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" event={"ID":"5dc2e621-86c7-4efa-801d-b31894185a08","Type":"ContainerDied","Data":"3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7"}
Jul 09 19:26:27.603248 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.603173    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b" event={"ID":"5dc2e621-86c7-4efa-801d-b31894185a08","Type":"ContainerDied","Data":"7953e49d220caaa23ca1e3a7722789e48a49aea76486c5141d46f5d3985b4ea9"}
Jul 09 19:26:27.603248 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.603194    2556 scope.go:117] "RemoveContainer" containerID="3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7"
Jul 09 19:26:27.613921 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.613895    2556 scope.go:117] "RemoveContainer" containerID="471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3"
Jul 09 19:26:27.622382 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.622362    2556 scope.go:117] "RemoveContainer" containerID="3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7"
Jul 09 19:26:27.622734 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:26:27.622712    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7\": container with ID starting with 3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7 not found: ID does not exist" containerID="3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7"
Jul 09 19:26:27.622836 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.622742    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7"} err="failed to get container status \"3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7\": rpc error: code = NotFound desc = could not find container \"3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7\": container with ID starting with 3afbbce10cc690907af7830441456ccda1f41171e9bf7d01daad5d6485232fd7 not found: ID does not exist"
Jul 09 19:26:27.622836 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.622768    2556 scope.go:117] "RemoveContainer" containerID="471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3"
Jul 09 19:26:27.623047 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:26:27.623003    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3\": container with ID starting with 471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3 not found: ID does not exist" containerID="471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3"
Jul 09 19:26:27.623117 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.623095    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3"} err="failed to get container status \"471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3\": rpc error: code = NotFound desc = could not find container \"471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3\": container with ID starting with 471cb841d0a0aeb449a36a32e852ecde99797eb16b3bbc883619c4b2bb6e4fa3 not found: ID does not exist"
Jul 09 19:26:27.625568 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.625547    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"]
Jul 09 19:26:27.628683 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:27.628659    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-eb3d8a-openshift-default-7544559645-wm67b"]
Jul 09 19:26:28.085339 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:28.085306    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" path="/var/lib/kubelet/pods/5dc2e621-86c7-4efa-801d-b31894185a08/volumes"
Jul 09 19:26:30.163228 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.163190    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:30.639672 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.639638    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:30.922847 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.922816    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"]
Jul 09 19:26:30.923436 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.923420    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy"
Jul 09 19:26:30.923507 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.923439    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy"
Jul 09 19:26:30.923507 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.923478    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" containerName="maas-api"
Jul 09 19:26:30.923507 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.923488    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" containerName="maas-api"
Jul 09 19:26:30.923630 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.923602    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy"
Jul 09 19:26:30.923630 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.923619    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3f5dbc4a-1d74-4f45-ac26-6ab3128ae6fb" containerName="maas-api"
Jul 09 19:26:30.928783 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.928761    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:30.931581 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.931552    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 19:26:30.931715 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.931673    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-c06f89-openshift-default-dockercfg-jkrvn\""
Jul 09 19:26:30.937651 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.937632    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"]
Jul 09 19:26:30.951141 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:30.951077    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:31.055805 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.055771    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfds2\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-kube-api-access-zfds2\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.055976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.055879    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-certs\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.055976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.055914    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-socket\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.055976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.055940    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-token\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.056149 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.055991    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-data\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.056149 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.056018    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-podinfo\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.056149 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.056041    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-envoy\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.056149 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.056080    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-credential-socket\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.056149 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.056132    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istiod-ca-cert\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157359 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157326    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-certs\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157359 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157371    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-socket\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157400    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-token\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157429    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-data\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157453    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-podinfo\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157476    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-envoy\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157502    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-credential-socket\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157570    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istiod-ca-cert\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.157662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157624    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfds2\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-kube-api-access-zfds2\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.158012 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.157895    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-socket\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.158169 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.158140    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-credential-socket\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.158245 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.158154    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-data\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.158245 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.158225    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-certs\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.158549 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.158512    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istiod-ca-cert\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.159960 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.159930    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-envoy\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.160049 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.160017    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-podinfo\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.165367 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.165345    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfds2\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-kube-api-access-zfds2\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.165823 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.165791    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-token\") pod \"e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") " pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.242290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.242204    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:31.293806 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.293762    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:31.388767 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.388694    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"]
Jul 09 19:26:31.391427 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:26:31.391400    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfac5b38a_154f_443c_ae94_c5b3aa5d4001.slice/crio-f437336a094695f21d66b9cab9297407f2a07b7aac8106e036a3b549991a6461 WatchSource:0}: Error finding container f437336a094695f21d66b9cab9297407f2a07b7aac8106e036a3b549991a6461: Status 404 returned error can't find the container with id f437336a094695f21d66b9cab9297407f2a07b7aac8106e036a3b549991a6461
Jul 09 19:26:31.394169 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.394134    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:26:31.394264 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.394202    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:26:31.394264 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.394229    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:26:31.633425 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.633324    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" event={"ID":"fac5b38a-154f-443c-ae94-c5b3aa5d4001","Type":"ContainerStarted","Data":"3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9"}
Jul 09 19:26:31.633425 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.633420    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" event={"ID":"fac5b38a-154f-443c-ae94-c5b3aa5d4001","Type":"ContainerStarted","Data":"f437336a094695f21d66b9cab9297407f2a07b7aac8106e036a3b549991a6461"}
Jul 09 19:26:31.652740 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:31.652671    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" podStartSLOduration=1.652650971 podStartE2EDuration="1.652650971s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:26:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:26:31.651602205 +0000 UTC m=+2002.004037267" watchObservedRunningTime="2026-07-09 19:26:31.652650971 +0000 UTC m=+2002.005086069"
Jul 09 19:26:32.242872 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:32.242840    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:32.248486 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:32.248460    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:32.637962 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:32.637875    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:32.638863 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:32.638840    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:33.576899 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.576858    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:33.637926 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.637893    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"]
Jul 09 19:26:33.638619 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.638564    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy"
Jul 09 19:26:33.638619 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.638588    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy"
Jul 09 19:26:33.638797 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.638772    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5dc2e621-86c7-4efa-801d-b31894185a08" containerName="istio-proxy"
Jul 09 19:26:33.642756 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.642661    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.645991 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.645967    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-c06f89-serving-cert\""
Jul 09 19:26:33.653763 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.653737    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"]
Jul 09 19:26:33.669476 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.669450    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:33.742179 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.742146    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:33.781286 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.781244    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.781459 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.781294    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a22582-7fb9-4ff8-919e-e14c2126fa14-openshift-service-ca\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.781522 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.781466    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j2vcd\" (UniqueName: \"kubernetes.io/projected/c5a22582-7fb9-4ff8-919e-e14c2126fa14-kube-api-access-j2vcd\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.882170 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.882075    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.882170 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.882111    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a22582-7fb9-4ff8-919e-e14c2126fa14-openshift-service-ca\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.882380 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.882173    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j2vcd\" (UniqueName: \"kubernetes.io/projected/c5a22582-7fb9-4ff8-919e-e14c2126fa14-kube-api-access-j2vcd\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.882380 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:26:33.882231    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-serving-cert: secret "maas-api-e2e-api-b-c06f89-serving-cert" not found
Jul 09 19:26:33.882380 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:26:33.882300    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls podName:c5a22582-7fb9-4ff8-919e-e14c2126fa14 nodeName:}" failed. No retries permitted until 2026-07-09 19:26:34.382284184 +0000 UTC m=+2004.734719227 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls") pod "maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx" (UID: "c5a22582-7fb9-4ff8-919e-e14c2126fa14") : secret "maas-api-e2e-api-b-c06f89-serving-cert" not found
Jul 09 19:26:33.882887 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.882868    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a22582-7fb9-4ff8-919e-e14c2126fa14-openshift-service-ca\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:33.892515 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:33.892492    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j2vcd\" (UniqueName: \"kubernetes.io/projected/c5a22582-7fb9-4ff8-919e-e14c2126fa14-kube-api-access-j2vcd\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:34.090315 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.090285    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:34.252513 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.252475    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:34.353324 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.353291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:34.387481 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.387449    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:34.389926 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.389900    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls\") pod \"maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:34.557839 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.557751    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:34.707744 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.707719    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"]
Jul 09 19:26:34.709674 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:26:34.709642    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5a22582_7fb9_4ff8_919e_e14c2126fa14.slice/crio-fa2917a6211d02f95b2eae3a56bcfaa1753a861241968d3688614836f3ede1fa WatchSource:0}: Error finding container fa2917a6211d02f95b2eae3a56bcfaa1753a861241968d3688614836f3ede1fa: Status 404 returned error can't find the container with id fa2917a6211d02f95b2eae3a56bcfaa1753a861241968d3688614836f3ede1fa
Jul 09 19:26:34.711229 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:34.711211    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:26:35.658229 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:35.658195    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx" event={"ID":"c5a22582-7fb9-4ff8-919e-e14c2126fa14","Type":"ContainerStarted","Data":"031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291"}
Jul 09 19:26:35.658419 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:35.658245    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx" event={"ID":"c5a22582-7fb9-4ff8-919e-e14c2126fa14","Type":"ContainerStarted","Data":"fa2917a6211d02f95b2eae3a56bcfaa1753a861241968d3688614836f3ede1fa"}
Jul 09 19:26:35.658419 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:35.658303    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:35.676299 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:35.676244    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx" podStartSLOduration=2.285760988 podStartE2EDuration="2.676223271s" totalImagesPullingTime="390.462283ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:26:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:26:35.674623375 +0000 UTC m=+2006.027058440" watchObservedRunningTime="2026-07-09 19:26:35.676223271 +0000 UTC m=+2006.028658337"
Jul 09 19:26:38.061215 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:38.061180    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:41.668464 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:41.668437    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:46.251753 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:46.251711    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:46.776446 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:46.776411    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:46.965890 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:46.965804    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:47.066855 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:47.066820    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:47.248013 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:47.247895    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:48.679698 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:48.679649    2556 patch_prober.go:28] interesting pod/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 09 19:26:48.680195 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:48.679724    2556 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 09 19:26:49.736814 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:49.736786    2556 generic.go:356] "Generic (PLEG): container finished" podID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerID="3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9" exitCode=0
Jul 09 19:26:49.737237 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:49.736865    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" event={"ID":"fac5b38a-154f-443c-ae94-c5b3aa5d4001","Type":"ContainerDied","Data":"3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9"}
Jul 09 19:26:49.737280 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:49.737236    2556 scope.go:117] "RemoveContainer" containerID="3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9"
Jul 09 19:26:49.738195 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:49.738158    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:26:49.738310 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:49.738225    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:26:49.738310 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:49.738255    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:26:50.744342 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:50.744306    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" event={"ID":"fac5b38a-154f-443c-ae94-c5b3aa5d4001","Type":"ContainerStarted","Data":"b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab"}
Jul 09 19:26:51.243364 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:51.243327    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:51.243603 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:51.243574    2556 patch_prober.go:28] interesting pod/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 09 19:26:51.243695 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:51.243632    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 09 19:26:52.242790 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:52.242755    2556 patch_prober.go:28] interesting pod/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 09 19:26:52.243204 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:52.242822    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 09 19:26:52.799007 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:52.798974    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:52.863232 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:52.863193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:53.079610 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:53.078258    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:53.242825 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:53.242788    2556 patch_prober.go:28] interesting pod/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 09 19:26:53.243228 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:53.242846    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 09 19:26:53.287258 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:53.287216    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:53.459759 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:53.459722    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:53.861159 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:53.861059    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:53.991315 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:53.991281    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:54.091981 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.091946    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:54.243548 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.243497    2556 patch_prober.go:28] interesting pod/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused" start-of-body=
Jul 09 19:26:54.244016 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.243580    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.62:15021/healthz/ready\": dial tcp 10.132.0.62:15021: connect: connection refused"
Jul 09 19:26:54.248170 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.248144    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"]
Jul 09 19:26:54.248389 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.248365    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx" podUID="c5a22582-7fb9-4ff8-919e-e14c2126fa14" containerName="maas-api" containerID="cri-o://031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291" gracePeriod=30
Jul 09 19:26:54.284923 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.284726    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:26:54.550704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.550667    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:54.584413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.584381    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls\") pod \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") "
Jul 09 19:26:54.584620 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.584458    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j2vcd\" (UniqueName: \"kubernetes.io/projected/c5a22582-7fb9-4ff8-919e-e14c2126fa14-kube-api-access-j2vcd\") pod \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") "
Jul 09 19:26:54.584620 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.584542    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a22582-7fb9-4ff8-919e-e14c2126fa14-openshift-service-ca\") pod \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\" (UID: \"c5a22582-7fb9-4ff8-919e-e14c2126fa14\") "
Jul 09 19:26:54.585148 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.585091    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5a22582-7fb9-4ff8-919e-e14c2126fa14-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c5a22582-7fb9-4ff8-919e-e14c2126fa14" (UID: "c5a22582-7fb9-4ff8-919e-e14c2126fa14"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:26:54.587659 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.587633    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5a22582-7fb9-4ff8-919e-e14c2126fa14-kube-api-access-j2vcd" (OuterVolumeSpecName: "kube-api-access-j2vcd") pod "c5a22582-7fb9-4ff8-919e-e14c2126fa14" (UID: "c5a22582-7fb9-4ff8-919e-e14c2126fa14"). InnerVolumeSpecName "kube-api-access-j2vcd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:26:54.587774 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.587637    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c5a22582-7fb9-4ff8-919e-e14c2126fa14" (UID: "c5a22582-7fb9-4ff8-919e-e14c2126fa14"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:26:54.685538 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.685490    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5a22582-7fb9-4ff8-919e-e14c2126fa14-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:54.685873 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.685553    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c5a22582-7fb9-4ff8-919e-e14c2126fa14-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:54.685873 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.685566    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j2vcd\" (UniqueName: \"kubernetes.io/projected/c5a22582-7fb9-4ff8-919e-e14c2126fa14-kube-api-access-j2vcd\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:26:54.773395 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.773355    2556 generic.go:356] "Generic (PLEG): container finished" podID="c5a22582-7fb9-4ff8-919e-e14c2126fa14" containerID="031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291" exitCode=0
Jul 09 19:26:54.773622 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.773417    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"
Jul 09 19:26:54.773622 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.773453    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx" event={"ID":"c5a22582-7fb9-4ff8-919e-e14c2126fa14","Type":"ContainerDied","Data":"031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291"}
Jul 09 19:26:54.773622 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.773573    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx" event={"ID":"c5a22582-7fb9-4ff8-919e-e14c2126fa14","Type":"ContainerDied","Data":"fa2917a6211d02f95b2eae3a56bcfaa1753a861241968d3688614836f3ede1fa"}
Jul 09 19:26:54.773622 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.773604    2556 scope.go:117] "RemoveContainer" containerID="031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291"
Jul 09 19:26:54.783820 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.783794    2556 scope.go:117] "RemoveContainer" containerID="031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291"
Jul 09 19:26:54.784080 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:26:54.784060    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291\": container with ID starting with 031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291 not found: ID does not exist" containerID="031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291"
Jul 09 19:26:54.784122 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.784090    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291"} err="failed to get container status \"031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291\": rpc error: code = NotFound desc = could not find container \"031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291\": container with ID starting with 031b3696df150add184f623c844a8f37589fe5228ff4537680bbf0e166962291 not found: ID does not exist"
Jul 09 19:26:54.796059 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.796026    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"]
Jul 09 19:26:54.802011 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:54.801989    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-c06f89-5d966ff4c-nkcmx"]
Jul 09 19:26:55.246874 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:55.246842    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:55.247273 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:55.247112    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:55.247848 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:55.247828    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:26:56.085110 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:26:56.085074    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5a22582-7fb9-4ff8-919e-e14c2126fa14" path="/var/lib/kubelet/pods/c5a22582-7fb9-4ff8-919e-e14c2126fa14/volumes"
Jul 09 19:27:00.283097 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:00.283054    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:00.346581 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:00.346546    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:00.526120 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:00.526075    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"]
Jul 09 19:27:00.526356 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:00.526332    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy" containerID="cri-o://b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab" gracePeriod=30
Jul 09 19:27:00.549721 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:00.549628    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:00.778607 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:00.778570    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:01.121670 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:01.121630    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:01.449986 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:01.449951    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:01.984107 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:01.984065    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:04.834783 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:04.834509    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:04.921755 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:04.921708    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:04.987764 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:04.987731    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:05.398061 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.398033    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:05.634216 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.634180    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:05.807920 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.807892    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:27:05.836947 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.836754    2556 generic.go:356] "Generic (PLEG): container finished" podID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerID="b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab" exitCode=0
Jul 09 19:27:05.836947 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.836840    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" event={"ID":"fac5b38a-154f-443c-ae94-c5b3aa5d4001","Type":"ContainerDied","Data":"b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab"}
Jul 09 19:27:05.836947 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.836908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv" event={"ID":"fac5b38a-154f-443c-ae94-c5b3aa5d4001","Type":"ContainerDied","Data":"f437336a094695f21d66b9cab9297407f2a07b7aac8106e036a3b549991a6461"}
Jul 09 19:27:05.836947 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.836921    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"
Jul 09 19:27:05.836947 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.836949    2556 scope.go:117] "RemoveContainer" containerID="b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab"
Jul 09 19:27:05.846831 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.846815    2556 scope.go:117] "RemoveContainer" containerID="3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9"
Jul 09 19:27:05.857567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.857546    2556 scope.go:117] "RemoveContainer" containerID="b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab"
Jul 09 19:27:05.857894 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:27:05.857868    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab\": container with ID starting with b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab not found: ID does not exist" containerID="b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab"
Jul 09 19:27:05.858001 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.857907    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab"} err="failed to get container status \"b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab\": rpc error: code = NotFound desc = could not find container \"b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab\": container with ID starting with b56c56c763d801322c6255cd6c5bc98a6228bba50bfa9b56820a5efecb1d3aab not found: ID does not exist"
Jul 09 19:27:05.858001 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.857932    2556 scope.go:117] "RemoveContainer" containerID="3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9"
Jul 09 19:27:05.858215 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:27:05.858196    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9\": container with ID starting with 3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9 not found: ID does not exist" containerID="3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9"
Jul 09 19:27:05.858270 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.858224    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9"} err="failed to get container status \"3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9\": rpc error: code = NotFound desc = could not find container \"3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9\": container with ID starting with 3298ec3be1fa2a16b7d3e8488ce0a2e49409822a9993e58de59c8444980b66c9 not found: ID does not exist"
Jul 09 19:27:05.891267 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891236    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-envoy\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891438 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891281    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-token\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891438 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891324    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zfds2\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-kube-api-access-zfds2\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891438 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891345    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-certs\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891438 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891370    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-socket\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891680 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891443    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-credential-socket\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891680 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891477    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istiod-ca-cert\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891680 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891514    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-data\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891680 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891559    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-podinfo\") pod \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\" (UID: \"fac5b38a-154f-443c-ae94-c5b3aa5d4001\") "
Jul 09 19:27:05.891887 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891735    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:05.891887 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891771    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:05.891995 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891894    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-certs\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.891995 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.891940    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-workload-socket\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.892173 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.892133    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:05.892311 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.892249    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:27:05.892938 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.892911    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-data" (OuterVolumeSpecName: "istio-data") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:05.893762 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.893740    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-token" (OuterVolumeSpecName: "istio-token") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:27:05.893932 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.893916    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:27:05.894178 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.894145    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-kube-api-access-zfds2" (OuterVolumeSpecName: "kube-api-access-zfds2") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "kube-api-access-zfds2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:27:05.894731 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.894716    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "fac5b38a-154f-443c-ae94-c5b3aa5d4001" (UID: "fac5b38a-154f-443c-ae94-c5b3aa5d4001"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 19:27:05.992586 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.992473    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfds2\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-kube-api-access-zfds2\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.992586 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.992507    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-credential-socket\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.992586 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.992522    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istiod-ca-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.992586 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.992554    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-data\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.992586 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.992568    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-podinfo\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.992586 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.992581    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-envoy\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:05.993029 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:05.992595    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fac5b38a-154f-443c-ae94-c5b3aa5d4001-istio-token\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:27:06.125477 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:06.125444    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:06.159354 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:06.159307    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"]
Jul 09 19:27:06.159923 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:06.159847    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-c06f89-openshift-default-6f66f94f89-mpdmv"]
Jul 09 19:27:08.085676 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:08.085632    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" path="/var/lib/kubelet/pods/fac5b38a-154f-443c-ae94-c5b3aa5d4001/volumes"
Jul 09 19:27:09.348132 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.348096    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:09.674638 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.674600    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:09.969514 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.969429    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"]
Jul 09 19:27:09.969884 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.969869    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5a22582-7fb9-4ff8-919e-e14c2126fa14" containerName="maas-api"
Jul 09 19:27:09.969946 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.969888    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5a22582-7fb9-4ff8-919e-e14c2126fa14" containerName="maas-api"
Jul 09 19:27:09.969946 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.969906    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy"
Jul 09 19:27:09.969946 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.969912    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy"
Jul 09 19:27:09.969946 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.969920    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy"
Jul 09 19:27:09.969946 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.969927    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy"
Jul 09 19:27:09.970134 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.970013    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy"
Jul 09 19:27:09.970134 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.970021    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="fac5b38a-154f-443c-ae94-c5b3aa5d4001" containerName="istio-proxy"
Jul 09 19:27:09.970134 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.970028    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5a22582-7fb9-4ff8-919e-e14c2126fa14" containerName="maas-api"
Jul 09 19:27:09.974686 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.974657    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:09.977466 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.977441    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 09 19:27:09.977697 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.977676    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-b20a51-openshift-default-dockercfg-sdjts\""
Jul 09 19:27:09.991216 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:09.991193    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"]
Jul 09 19:27:10.019413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.019384    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:10.131541 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131496    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-data\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131541 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131545    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-envoy\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131755 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131611    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-socket\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131755 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131683    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tmms\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-kube-api-access-5tmms\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131755 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131736    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-credential-socket\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131871 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131764    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-certs\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131871 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131788    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-token\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131871 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131816    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-podinfo\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.131997 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.131897    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istiod-ca-cert\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233063 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.232981    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-certs\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233063 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233029    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-token\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233063 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233059    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-podinfo\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233089    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istiod-ca-cert\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233157    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-data\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233203    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-envoy\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233233    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-socket\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233280    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tmms\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-kube-api-access-5tmms\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233379 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233326    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-credential-socket\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233736 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233636    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-data\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233736 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233664    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-credential-socket\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233859 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233748    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-certs\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233859 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233775    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-socket\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.233964 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.233928    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istiod-ca-cert\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.235478 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.235459    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-envoy\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.235750 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.235732    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-podinfo\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.241650 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.241629    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-token\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.241902 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.241870    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tmms\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-kube-api-access-5tmms\") pod \"e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") " pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.267318 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.267291    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:10.285394 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.285363    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:10.435972 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.435939    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"]
Jul 09 19:27:10.439384 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:27:10.439355    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280a1a95_e8b1_4c52_822f_8c4a07c900bb.slice/crio-a7efb900860e13177eaf5cc31d37115064a9d3323d6bbfc0713c5c0a70a66e31 WatchSource:0}: Error finding container a7efb900860e13177eaf5cc31d37115064a9d3323d6bbfc0713c5c0a70a66e31: Status 404 returned error can't find the container with id a7efb900860e13177eaf5cc31d37115064a9d3323d6bbfc0713c5c0a70a66e31
Jul 09 19:27:10.441655 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.441611    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:27:10.441755 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.441708    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:27:10.441796 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.441750    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:27:10.865488 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.865394    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" event={"ID":"280a1a95-e8b1-4c52-822f-8c4a07c900bb","Type":"ContainerStarted","Data":"823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99"}
Jul 09 19:27:10.865488 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.865451    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" event={"ID":"280a1a95-e8b1-4c52-822f-8c4a07c900bb","Type":"ContainerStarted","Data":"a7efb900860e13177eaf5cc31d37115064a9d3323d6bbfc0713c5c0a70a66e31"}
Jul 09 19:27:10.886838 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:10.886785    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" podStartSLOduration=1.886769895 podStartE2EDuration="1.886769895s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:10.886125639 +0000 UTC m=+2041.238560715" watchObservedRunningTime="2026-07-09 19:27:10.886769895 +0000 UTC m=+2041.239204959"
Jul 09 19:27:11.286117 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:11.286080    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:11.290819 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:11.290793    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:11.870232 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:11.870182    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:11.871165 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:11.871147    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:12.786781 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:12.786751    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:12.885032 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:12.884982    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:13.031437 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.031403    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"]
Jul 09 19:27:13.035275 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.035248    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.038459 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.038389    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-b20a51-serving-cert\""
Jul 09 19:27:13.052005 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.051976    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"]
Jul 09 19:27:13.117659 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.117628    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:13.163647 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.163607    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.163837 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.163666    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjhjx\" (UniqueName: \"kubernetes.io/projected/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-kube-api-access-kjhjx\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.164083 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.164061    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.265036 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.264993    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.265220 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.265083    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.265220 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.265129    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjhjx\" (UniqueName: \"kubernetes.io/projected/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-kube-api-access-kjhjx\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.265220 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:27:13.265153    2556 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-serving-cert: secret "maas-api-e2e-shared-b-b20a51-serving-cert" not found
Jul 09 19:27:13.265415 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:27:13.265222    2556 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls podName:3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7 nodeName:}" failed. No retries permitted until 2026-07-09 19:27:13.765199739 +0000 UTC m=+2044.117634808 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls") pod "maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn" (UID: "3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7") : secret "maas-api-e2e-shared-b-b20a51-serving-cert" not found
Jul 09 19:27:13.265847 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.265826    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-openshift-service-ca\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.274298 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.274277    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjhjx\" (UniqueName: \"kubernetes.io/projected/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-kube-api-access-kjhjx\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.534402 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.534367    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:13.634364 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.634324    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:13.734988 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.734947    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:13.770609 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.770570    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.773201 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.773176    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls\") pod \"maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:13.947772 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:13.947732    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:14.299071 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:14.299046    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"]
Jul 09 19:27:14.301695 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:27:14.301670    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c3dd1bf_d46a_48c9_a293_eef8a24f4cb7.slice/crio-685559a39de4afb9b82596800a4989d669d1a1a0626de448e693959bd9d9150d WatchSource:0}: Error finding container 685559a39de4afb9b82596800a4989d669d1a1a0626de448e693959bd9d9150d: Status 404 returned error can't find the container with id 685559a39de4afb9b82596800a4989d669d1a1a0626de448e693959bd9d9150d
Jul 09 19:27:14.889745 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:14.889704    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn" event={"ID":"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7","Type":"ContainerStarted","Data":"f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39"}
Jul 09 19:27:14.889941 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:14.889774    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:14.889941 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:14.889797    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn" event={"ID":"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7","Type":"ContainerStarted","Data":"685559a39de4afb9b82596800a4989d669d1a1a0626de448e693959bd9d9150d"}
Jul 09 19:27:14.912588 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:14.912535    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn" podStartSLOduration=1.549828061 podStartE2EDuration="1.912503104s" totalImagesPullingTime="362.675043ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:27:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:27:14.909801842 +0000 UTC m=+2045.262236909" watchObservedRunningTime="2026-07-09 19:27:14.912503104 +0000 UTC m=+2045.264938174"
Jul 09 19:27:17.292350 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:17.292311    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:20.901768 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:20.901730    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:27:22.118913 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:22.118821    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:24.942166 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:24.942135    2556 generic.go:356] "Generic (PLEG): container finished" podID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerID="823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99" exitCode=0
Jul 09 19:27:24.942664 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:24.942206    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" event={"ID":"280a1a95-e8b1-4c52-822f-8c4a07c900bb","Type":"ContainerDied","Data":"823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99"}
Jul 09 19:27:24.942664 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:24.942630    2556 scope.go:117] "RemoveContainer" containerID="823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99"
Jul 09 19:27:24.943656 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:24.943629    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:27:24.943752 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:24.943694    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:27:24.943752 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:24.943724    2556 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236232Ki","pods":"250"}
Jul 09 19:27:25.950136 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:25.950099    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" event={"ID":"280a1a95-e8b1-4c52-822f-8c4a07c900bb","Type":"ContainerStarted","Data":"2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d"}
Jul 09 19:27:26.128854 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:26.128814    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:26.229203 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:26.229108    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:26.286265 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:26.286223    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:26.287655 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:26.287626    2556 patch_prober.go:28] interesting pod/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 09 19:27:26.288243 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:26.288205    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 09 19:27:26.872552 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:26.870718    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:27.286446 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:27.286409    2556 patch_prober.go:28] interesting pod/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 09 19:27:27.287077 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:27.287044    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 09 19:27:28.286251 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:28.286214    2556 patch_prober.go:28] interesting pod/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 09 19:27:28.286581 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:28.286549    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 09 19:27:29.286551 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:29.286497    2556 patch_prober.go:28] interesting pod/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 09 19:27:29.286780 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:29.286583    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 09 19:27:30.286154 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:30.286104    2556 patch_prober.go:28] interesting pod/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Jul 09 19:27:30.286375 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:30.286177    2556 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Jul 09 19:27:31.290186 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:31.290157    2556 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:31.291242 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:31.291222    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:27:34.271985 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:34.271908    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:34.372227 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:34.372194    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:27:56.683487 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:27:56.683456    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:28:04.115695 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:28:04.115631    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:28:10.200549 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:28:10.200498    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:28:10.205909 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:28:10.205885    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:28:10.206937 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:28:10.206915    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:28:10.211922 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:28:10.211905    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:28:20.378414 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:28:20.378371    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:30:00.139712 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.139677    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"]
Jul 09 19:30:00.143332 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.143312    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"]
Jul 09 19:30:00.143408 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.143391    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"
Jul 09 19:30:00.146604 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.146583    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"]
Jul 09 19:30:00.146756 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.146694    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"
Jul 09 19:30:00.149953 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.149935    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"
Jul 09 19:30:00.154949 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.154925    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"]
Jul 09 19:30:00.157866 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.157842    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"]
Jul 09 19:30:00.160794 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.160768    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"]
Jul 09 19:30:00.210099 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.210059    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxkxf\" (UniqueName: \"kubernetes.io/projected/5cb82c1c-3443-405a-98c7-7d753063d275-kube-api-access-nxkxf\") pod \"maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78\" (UID: \"5cb82c1c-3443-405a-98c7-7d753063d275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"
Jul 09 19:30:00.210332 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.210120    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tl2kw\" (UniqueName: \"kubernetes.io/projected/1ce193b3-117e-4450-b3ff-2ba2c76937e5-kube-api-access-tl2kw\") pod \"maas-api-key-cleanup-29727090-6p28x\" (UID: \"1ce193b3-117e-4450-b3ff-2ba2c76937e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"
Jul 09 19:30:00.210332 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.210159    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n62rw\" (UniqueName: \"kubernetes.io/projected/9b20aba8-8106-4e99-af70-cec6aff1cfde-kube-api-access-n62rw\") pod \"maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77\" (UID: \"9b20aba8-8106-4e99-af70-cec6aff1cfde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"
Jul 09 19:30:00.311377 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.311339    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n62rw\" (UniqueName: \"kubernetes.io/projected/9b20aba8-8106-4e99-af70-cec6aff1cfde-kube-api-access-n62rw\") pod \"maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77\" (UID: \"9b20aba8-8106-4e99-af70-cec6aff1cfde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"
Jul 09 19:30:00.311589 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.311402    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxkxf\" (UniqueName: \"kubernetes.io/projected/5cb82c1c-3443-405a-98c7-7d753063d275-kube-api-access-nxkxf\") pod \"maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78\" (UID: \"5cb82c1c-3443-405a-98c7-7d753063d275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"
Jul 09 19:30:00.311589 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.311453    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tl2kw\" (UniqueName: \"kubernetes.io/projected/1ce193b3-117e-4450-b3ff-2ba2c76937e5-kube-api-access-tl2kw\") pod \"maas-api-key-cleanup-29727090-6p28x\" (UID: \"1ce193b3-117e-4450-b3ff-2ba2c76937e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"
Jul 09 19:30:00.320576 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.320522    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxkxf\" (UniqueName: \"kubernetes.io/projected/5cb82c1c-3443-405a-98c7-7d753063d275-kube-api-access-nxkxf\") pod \"maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78\" (UID: \"5cb82c1c-3443-405a-98c7-7d753063d275\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"
Jul 09 19:30:00.320707 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.320581    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tl2kw\" (UniqueName: \"kubernetes.io/projected/1ce193b3-117e-4450-b3ff-2ba2c76937e5-kube-api-access-tl2kw\") pod \"maas-api-key-cleanup-29727090-6p28x\" (UID: \"1ce193b3-117e-4450-b3ff-2ba2c76937e5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"
Jul 09 19:30:00.320764 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.320747    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n62rw\" (UniqueName: \"kubernetes.io/projected/9b20aba8-8106-4e99-af70-cec6aff1cfde-kube-api-access-n62rw\") pod \"maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77\" (UID: \"9b20aba8-8106-4e99-af70-cec6aff1cfde\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"
Jul 09 19:30:00.456211 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.456164    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"
Jul 09 19:30:00.466471 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.466433    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"
Jul 09 19:30:00.474630 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.474604    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"
Jul 09 19:30:00.630358 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.630299    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"]
Jul 09 19:30:00.631945 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:30:00.631920    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb82c1c_3443_405a_98c7_7d753063d275.slice/crio-d786c71203ac818785dacb2e2499d961d161f2c2b5ef2ad8d5f7f82a2b57b2e8 WatchSource:0}: Error finding container d786c71203ac818785dacb2e2499d961d161f2c2b5ef2ad8d5f7f82a2b57b2e8: Status 404 returned error can't find the container with id d786c71203ac818785dacb2e2499d961d161f2c2b5ef2ad8d5f7f82a2b57b2e8
Jul 09 19:30:00.786679 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.786576    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78" event={"ID":"5cb82c1c-3443-405a-98c7-7d753063d275","Type":"ContainerStarted","Data":"d786c71203ac818785dacb2e2499d961d161f2c2b5ef2ad8d5f7f82a2b57b2e8"}
Jul 09 19:30:00.875823 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.875740    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"]
Jul 09 19:30:00.877989 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:00.877963    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"]
Jul 09 19:30:00.881883 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:30:00.881855    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b20aba8_8106_4e99_af70_cec6aff1cfde.slice/crio-c5d86744f3afe4a65cbabdedfbae7b18ddc50b03bfa1a71d945a1632c6845310 WatchSource:0}: Error finding container c5d86744f3afe4a65cbabdedfbae7b18ddc50b03bfa1a71d945a1632c6845310: Status 404 returned error can't find the container with id c5d86744f3afe4a65cbabdedfbae7b18ddc50b03bfa1a71d945a1632c6845310
Jul 09 19:30:00.882053 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:30:00.882035    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ce193b3_117e_4450_b3ff_2ba2c76937e5.slice/crio-f579a8eadc766648d8eb64dee9340370c3521a311f45934bf695674415b060a8 WatchSource:0}: Error finding container f579a8eadc766648d8eb64dee9340370c3521a311f45934bf695674415b060a8: Status 404 returned error can't find the container with id f579a8eadc766648d8eb64dee9340370c3521a311f45934bf695674415b060a8
Jul 09 19:30:01.793509 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.793467    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x" event={"ID":"1ce193b3-117e-4450-b3ff-2ba2c76937e5","Type":"ContainerStarted","Data":"86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263"}
Jul 09 19:30:01.793996 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.793556    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x" event={"ID":"1ce193b3-117e-4450-b3ff-2ba2c76937e5","Type":"ContainerStarted","Data":"f579a8eadc766648d8eb64dee9340370c3521a311f45934bf695674415b060a8"}
Jul 09 19:30:01.795127 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.795097    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77" event={"ID":"9b20aba8-8106-4e99-af70-cec6aff1cfde","Type":"ContainerStarted","Data":"db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df"}
Jul 09 19:30:01.795275 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.795145    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77" event={"ID":"9b20aba8-8106-4e99-af70-cec6aff1cfde","Type":"ContainerStarted","Data":"c5d86744f3afe4a65cbabdedfbae7b18ddc50b03bfa1a71d945a1632c6845310"}
Jul 09 19:30:01.796688 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.796664    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78" event={"ID":"5cb82c1c-3443-405a-98c7-7d753063d275","Type":"ContainerStarted","Data":"92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad"}
Jul 09 19:30:01.809263 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.809203    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x" podStartSLOduration=1.809180911 podStartE2EDuration="1.809180911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:30:01.808144025 +0000 UTC m=+2212.160579084" watchObservedRunningTime="2026-07-09 19:30:01.809180911 +0000 UTC m=+2212.161615977"
Jul 09 19:30:01.824131 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.824082    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77" podStartSLOduration=1.824066997 podStartE2EDuration="1.824066997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:30:01.822715518 +0000 UTC m=+2212.175150584" watchObservedRunningTime="2026-07-09 19:30:01.824066997 +0000 UTC m=+2212.176502062"
Jul 09 19:30:01.838217 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:01.838166    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78" podStartSLOduration=1.8381503019999998 podStartE2EDuration="1.838150302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:30:01.837833046 +0000 UTC m=+2212.190268125" watchObservedRunningTime="2026-07-09 19:30:01.838150302 +0000 UTC m=+2212.190585368"
Jul 09 19:30:24.372084 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:24.372052    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:30:25.549670 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:25.549639    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:30:36.272003 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:30:36.271912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:31:00.011488 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:00.011452    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"]
Jul 09 19:31:00.011893 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:00.011670    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x" podUID="1ce193b3-117e-4450-b3ff-2ba2c76937e5" containerName="cleanup" containerID="cri-o://86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263" gracePeriod=30
Jul 09 19:31:00.015663 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:00.015630    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"]
Jul 09 19:31:00.015876 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:00.015839    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77" podUID="9b20aba8-8106-4e99-af70-cec6aff1cfde" containerName="cleanup" containerID="cri-o://db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df" gracePeriod=30
Jul 09 19:31:00.017740 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:00.017716    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"]
Jul 09 19:31:00.017949 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:00.017928    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78" podUID="5cb82c1c-3443-405a-98c7-7d753063d275" containerName="cleanup" containerID="cri-o://92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad" gracePeriod=30
Jul 09 19:31:08.075259 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:08.075193    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:31:21.902968 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:21.902912    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:31:22.003818 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:22.003787    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:31:30.244510 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.244479    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"
Jul 09 19:31:30.272004 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.271982    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"
Jul 09 19:31:30.279638 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.279608    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"
Jul 09 19:31:30.294460 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.294421    2556 generic.go:356] "Generic (PLEG): container finished" podID="9b20aba8-8106-4e99-af70-cec6aff1cfde" containerID="db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df" exitCode=137
Jul 09 19:31:30.294460 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.294457    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"
Jul 09 19:31:30.294771 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.294505    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77" event={"ID":"9b20aba8-8106-4e99-af70-cec6aff1cfde","Type":"ContainerDied","Data":"db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df"}
Jul 09 19:31:30.294771 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.294597    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77" event={"ID":"9b20aba8-8106-4e99-af70-cec6aff1cfde","Type":"ContainerDied","Data":"c5d86744f3afe4a65cbabdedfbae7b18ddc50b03bfa1a71d945a1632c6845310"}
Jul 09 19:31:30.294771 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.294641    2556 scope.go:117] "RemoveContainer" containerID="db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df"
Jul 09 19:31:30.296108 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.296083    2556 generic.go:356] "Generic (PLEG): container finished" podID="5cb82c1c-3443-405a-98c7-7d753063d275" containerID="92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad" exitCode=137
Jul 09 19:31:30.296208 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.296143    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"
Jul 09 19:31:30.296208 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.296164    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78" event={"ID":"5cb82c1c-3443-405a-98c7-7d753063d275","Type":"ContainerDied","Data":"92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad"}
Jul 09 19:31:30.296328 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.296219    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78" event={"ID":"5cb82c1c-3443-405a-98c7-7d753063d275","Type":"ContainerDied","Data":"d786c71203ac818785dacb2e2499d961d161f2c2b5ef2ad8d5f7f82a2b57b2e8"}
Jul 09 19:31:30.297772 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.297744    2556 generic.go:356] "Generic (PLEG): container finished" podID="1ce193b3-117e-4450-b3ff-2ba2c76937e5" containerID="86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263" exitCode=137
Jul 09 19:31:30.297893 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.297777    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x" event={"ID":"1ce193b3-117e-4450-b3ff-2ba2c76937e5","Type":"ContainerDied","Data":"86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263"}
Jul 09 19:31:30.297893 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.297802    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"
Jul 09 19:31:30.297893 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.297817    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x" event={"ID":"1ce193b3-117e-4450-b3ff-2ba2c76937e5","Type":"ContainerDied","Data":"f579a8eadc766648d8eb64dee9340370c3521a311f45934bf695674415b060a8"}
Jul 09 19:31:30.305062 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.305040    2556 scope.go:117] "RemoveContainer" containerID="db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df"
Jul 09 19:31:30.305391 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:31:30.305366    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df\": container with ID starting with db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df not found: ID does not exist" containerID="db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df"
Jul 09 19:31:30.305471 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.305402    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df"} err="failed to get container status \"db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df\": rpc error: code = NotFound desc = could not find container \"db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df\": container with ID starting with db832c5465e2b9ef91253354dcf3d5113b738444b9303614014fd58c309d25df not found: ID does not exist"
Jul 09 19:31:30.305471 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.305429    2556 scope.go:117] "RemoveContainer" containerID="92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad"
Jul 09 19:31:30.318676 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.318649    2556 scope.go:117] "RemoveContainer" containerID="92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad"
Jul 09 19:31:30.319008 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:31:30.318991    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad\": container with ID starting with 92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad not found: ID does not exist" containerID="92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad"
Jul 09 19:31:30.319078 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.319017    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad"} err="failed to get container status \"92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad\": rpc error: code = NotFound desc = could not find container \"92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad\": container with ID starting with 92020af3b71b368bb610da4129359ab0b42d145d493c591ee1d6d3a76c8ec9ad not found: ID does not exist"
Jul 09 19:31:30.319078 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.319054    2556 scope.go:117] "RemoveContainer" containerID="86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263"
Jul 09 19:31:30.331392 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.331362    2556 scope.go:117] "RemoveContainer" containerID="86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263"
Jul 09 19:31:30.331909 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:31:30.331882    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263\": container with ID starting with 86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263 not found: ID does not exist" containerID="86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263"
Jul 09 19:31:30.332025 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.331918    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263"} err="failed to get container status \"86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263\": rpc error: code = NotFound desc = could not find container \"86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263\": container with ID starting with 86b35826c1a4a6b642f4f275d90b3ef93befe65a13cd7ac883eb26cf0cf02263 not found: ID does not exist"
Jul 09 19:31:30.358932 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.358901    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n62rw\" (UniqueName: \"kubernetes.io/projected/9b20aba8-8106-4e99-af70-cec6aff1cfde-kube-api-access-n62rw\") pod \"9b20aba8-8106-4e99-af70-cec6aff1cfde\" (UID: \"9b20aba8-8106-4e99-af70-cec6aff1cfde\") "
Jul 09 19:31:30.359155 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.358977    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxkxf\" (UniqueName: \"kubernetes.io/projected/5cb82c1c-3443-405a-98c7-7d753063d275-kube-api-access-nxkxf\") pod \"5cb82c1c-3443-405a-98c7-7d753063d275\" (UID: \"5cb82c1c-3443-405a-98c7-7d753063d275\") "
Jul 09 19:31:30.359155 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.359108    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tl2kw\" (UniqueName: \"kubernetes.io/projected/1ce193b3-117e-4450-b3ff-2ba2c76937e5-kube-api-access-tl2kw\") pod \"1ce193b3-117e-4450-b3ff-2ba2c76937e5\" (UID: \"1ce193b3-117e-4450-b3ff-2ba2c76937e5\") "
Jul 09 19:31:30.361368 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.361333    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ce193b3-117e-4450-b3ff-2ba2c76937e5-kube-api-access-tl2kw" (OuterVolumeSpecName: "kube-api-access-tl2kw") pod "1ce193b3-117e-4450-b3ff-2ba2c76937e5" (UID: "1ce193b3-117e-4450-b3ff-2ba2c76937e5"). InnerVolumeSpecName "kube-api-access-tl2kw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:31:30.361545 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.361372    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb82c1c-3443-405a-98c7-7d753063d275-kube-api-access-nxkxf" (OuterVolumeSpecName: "kube-api-access-nxkxf") pod "5cb82c1c-3443-405a-98c7-7d753063d275" (UID: "5cb82c1c-3443-405a-98c7-7d753063d275"). InnerVolumeSpecName "kube-api-access-nxkxf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:31:30.361675 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.361658    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b20aba8-8106-4e99-af70-cec6aff1cfde-kube-api-access-n62rw" (OuterVolumeSpecName: "kube-api-access-n62rw") pod "9b20aba8-8106-4e99-af70-cec6aff1cfde" (UID: "9b20aba8-8106-4e99-af70-cec6aff1cfde"). InnerVolumeSpecName "kube-api-access-n62rw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:31:30.460290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.460245    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n62rw\" (UniqueName: \"kubernetes.io/projected/9b20aba8-8106-4e99-af70-cec6aff1cfde-kube-api-access-n62rw\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:31:30.460290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.460282    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxkxf\" (UniqueName: \"kubernetes.io/projected/5cb82c1c-3443-405a-98c7-7d753063d275-kube-api-access-nxkxf\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:31:30.460290 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.460295    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tl2kw\" (UniqueName: \"kubernetes.io/projected/1ce193b3-117e-4450-b3ff-2ba2c76937e5-kube-api-access-tl2kw\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:31:30.623446 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.623407    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"]
Jul 09 19:31:30.629825 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.629776    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-b20a51-29727090-4rj77"]
Jul 09 19:31:30.642939 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.642898    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"]
Jul 09 19:31:30.650803 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.650774    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-5d8f8f-29727090-lxh78"]
Jul 09 19:31:30.663060 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.663022    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"]
Jul 09 19:31:30.668048 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:30.668018    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29727090-6p28x"]
Jul 09 19:31:32.085761 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:32.085726    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1ce193b3-117e-4450-b3ff-2ba2c76937e5" path="/var/lib/kubelet/pods/1ce193b3-117e-4450-b3ff-2ba2c76937e5/volumes"
Jul 09 19:31:32.086146 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:32.086065    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5cb82c1c-3443-405a-98c7-7d753063d275" path="/var/lib/kubelet/pods/5cb82c1c-3443-405a-98c7-7d753063d275/volumes"
Jul 09 19:31:32.086362 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:32.086347    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b20aba8-8106-4e99-af70-cec6aff1cfde" path="/var/lib/kubelet/pods/9b20aba8-8106-4e99-af70-cec6aff1cfde/volumes"
Jul 09 19:31:44.015710 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:31:44.015664    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:00.360050 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:00.360011    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:01.157363 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:01.157325    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:12.378419 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:12.378388    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:26.888679 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:26.888623    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:26.986205 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:26.986161    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:34.998976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:34.998941    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:35.099934 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:35.099891    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:32:45.102146 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:32:45.102103    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:10.232803 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:10.232774    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:33:10.241233 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:10.241199    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:33:10.244033 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:10.244014    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:33:10.249020 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:10.249002    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-kbcj5_5f6e8995-b752-4679-887f-4cd3a9826b7c/ovn-acl-logging/0.log"
Jul 09 19:33:32.677601 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:32.677490    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:33.920199 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:33.920161    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:34.301176 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301089    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7482-140307190337968"]
Jul 09 19:33:34.301575 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301555    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1ce193b3-117e-4450-b3ff-2ba2c76937e5" containerName="cleanup"
Jul 09 19:33:34.301575 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301570    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="1ce193b3-117e-4450-b3ff-2ba2c76937e5" containerName="cleanup"
Jul 09 19:33:34.301690 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301605    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5cb82c1c-3443-405a-98c7-7d753063d275" containerName="cleanup"
Jul 09 19:33:34.301690 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301611    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="5cb82c1c-3443-405a-98c7-7d753063d275" containerName="cleanup"
Jul 09 19:33:34.301690 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301619    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b20aba8-8106-4e99-af70-cec6aff1cfde" containerName="cleanup"
Jul 09 19:33:34.301690 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301626    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b20aba8-8106-4e99-af70-cec6aff1cfde" containerName="cleanup"
Jul 09 19:33:34.301811 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301699    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="5cb82c1c-3443-405a-98c7-7d753063d275" containerName="cleanup"
Jul 09 19:33:34.301811 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301708    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b20aba8-8106-4e99-af70-cec6aff1cfde" containerName="cleanup"
Jul 09 19:33:34.301811 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.301717    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="1ce193b3-117e-4450-b3ff-2ba2c76937e5" containerName="cleanup"
Jul 09 19:33:34.304790 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.304775    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:34.307782 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.307764    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-7trn4\""
Jul 09 19:33:34.312042 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.311835    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307190337968"]
Jul 09 19:33:34.374332 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.374302    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zckx7\" (UniqueName: \"kubernetes.io/projected/23f4ab2f-43a6-413b-9250-7ad0e808bda7-kube-api-access-zckx7\") pod \"test-curl-7482-140307190337968\" (UID: \"23f4ab2f-43a6-413b-9250-7ad0e808bda7\") " pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:34.475296 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.475255    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zckx7\" (UniqueName: \"kubernetes.io/projected/23f4ab2f-43a6-413b-9250-7ad0e808bda7-kube-api-access-zckx7\") pod \"test-curl-7482-140307190337968\" (UID: \"23f4ab2f-43a6-413b-9250-7ad0e808bda7\") " pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:34.484853 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.484823    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zckx7\" (UniqueName: \"kubernetes.io/projected/23f4ab2f-43a6-413b-9250-7ad0e808bda7-kube-api-access-zckx7\") pod \"test-curl-7482-140307190337968\" (UID: \"23f4ab2f-43a6-413b-9250-7ad0e808bda7\") " pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:34.615178 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.615080    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:34.762370 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.762341    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307190337968"]
Jul 09 19:33:34.764121 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:33:34.764096    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23f4ab2f_43a6_413b_9250_7ad0e808bda7.slice/crio-ac58b6ae412e63090963cdcd7504492a45fd1398a278e854b8f9750ac1d33770 WatchSource:0}: Error finding container ac58b6ae412e63090963cdcd7504492a45fd1398a278e854b8f9750ac1d33770: Status 404 returned error can't find the container with id ac58b6ae412e63090963cdcd7504492a45fd1398a278e854b8f9750ac1d33770
Jul 09 19:33:34.765326 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.765307    2556 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 09 19:33:34.945250 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:34.945211    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307190337968" event={"ID":"23f4ab2f-43a6-413b-9250-7ad0e808bda7","Type":"ContainerStarted","Data":"ac58b6ae412e63090963cdcd7504492a45fd1398a278e854b8f9750ac1d33770"}
Jul 09 19:33:35.952085 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:35.952048    2556 generic.go:356] "Generic (PLEG): container finished" podID="23f4ab2f-43a6-413b-9250-7ad0e808bda7" containerID="3e3e0dfe075417100bdf653da405645034af40b9b6fe4a57916ffc773b9b322a" exitCode=0
Jul 09 19:33:35.952564 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:35.952104    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307190337968" event={"ID":"23f4ab2f-43a6-413b-9250-7ad0e808bda7","Type":"ContainerDied","Data":"3e3e0dfe075417100bdf653da405645034af40b9b6fe4a57916ffc773b9b322a"}
Jul 09 19:33:37.100961 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.100937    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:37.149440 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.149407    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7482-140307190337968_23f4ab2f-43a6-413b-9250-7ad0e808bda7/test-curl-7482-140307190337968/0.log"
Jul 09 19:33:37.195010 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.194976    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7482-140307190337968"]
Jul 09 19:33:37.199095 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.199070    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zckx7\" (UniqueName: \"kubernetes.io/projected/23f4ab2f-43a6-413b-9250-7ad0e808bda7-kube-api-access-zckx7\") pod \"23f4ab2f-43a6-413b-9250-7ad0e808bda7\" (UID: \"23f4ab2f-43a6-413b-9250-7ad0e808bda7\") "
Jul 09 19:33:37.200211 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.200186    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7482-140307190337968"]
Jul 09 19:33:37.201185 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.201163    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f4ab2f-43a6-413b-9250-7ad0e808bda7-kube-api-access-zckx7" (OuterVolumeSpecName: "kube-api-access-zckx7") pod "23f4ab2f-43a6-413b-9250-7ad0e808bda7" (UID: "23f4ab2f-43a6-413b-9250-7ad0e808bda7"). InnerVolumeSpecName "kube-api-access-zckx7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:33:37.300154 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.300071    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zckx7\" (UniqueName: \"kubernetes.io/projected/23f4ab2f-43a6-413b-9250-7ad0e808bda7-kube-api-access-zckx7\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:33:37.341153 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.341119    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7482-140307190338480"]
Jul 09 19:33:37.341562 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.341548    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23f4ab2f-43a6-413b-9250-7ad0e808bda7" containerName="test-curl-7482-140307190337968"
Jul 09 19:33:37.341622 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.341563    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="23f4ab2f-43a6-413b-9250-7ad0e808bda7" containerName="test-curl-7482-140307190337968"
Jul 09 19:33:37.341660 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.341650    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="23f4ab2f-43a6-413b-9250-7ad0e808bda7" containerName="test-curl-7482-140307190337968"
Jul 09 19:33:37.344911 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.344896    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:37.363218 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.363195    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307190338480"]
Jul 09 19:33:37.502104 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.502065    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsh5p\" (UniqueName: \"kubernetes.io/projected/3bf69fd5-5d2c-4437-891d-57f489765ea5-kube-api-access-nsh5p\") pod \"test-curl-7482-140307190338480\" (UID: \"3bf69fd5-5d2c-4437-891d-57f489765ea5\") " pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:37.603161 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.603074    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsh5p\" (UniqueName: \"kubernetes.io/projected/3bf69fd5-5d2c-4437-891d-57f489765ea5-kube-api-access-nsh5p\") pod \"test-curl-7482-140307190338480\" (UID: \"3bf69fd5-5d2c-4437-891d-57f489765ea5\") " pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:37.627502 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.627473    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsh5p\" (UniqueName: \"kubernetes.io/projected/3bf69fd5-5d2c-4437-891d-57f489765ea5-kube-api-access-nsh5p\") pod \"test-curl-7482-140307190338480\" (UID: \"3bf69fd5-5d2c-4437-891d-57f489765ea5\") " pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:37.653882 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.653854    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:37.802746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.802723    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307190338480"]
Jul 09 19:33:37.804865 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:33:37.804836    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3bf69fd5_5d2c_4437_891d_57f489765ea5.slice/crio-781b8134c735310d9445b88b0d5d596aa73bbd4bd2add18c450ff1ea9852a415 WatchSource:0}: Error finding container 781b8134c735310d9445b88b0d5d596aa73bbd4bd2add18c450ff1ea9852a415: Status 404 returned error can't find the container with id 781b8134c735310d9445b88b0d5d596aa73bbd4bd2add18c450ff1ea9852a415
Jul 09 19:33:37.965197 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.965160    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307190338480" event={"ID":"3bf69fd5-5d2c-4437-891d-57f489765ea5","Type":"ContainerStarted","Data":"781b8134c735310d9445b88b0d5d596aa73bbd4bd2add18c450ff1ea9852a415"}
Jul 09 19:33:37.966426 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.966408    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:37.966426 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:37.966414    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac58b6ae412e63090963cdcd7504492a45fd1398a278e854b8f9750ac1d33770"
Jul 09 19:33:37.978306 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:33:37.978279    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307190337968\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="23f4ab2f-43a6-413b-9250-7ad0e808bda7" pod="openshift-ingress/test-curl-7482-140307190337968"
Jul 09 19:33:38.085738 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:38.085698    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23f4ab2f-43a6-413b-9250-7ad0e808bda7" path="/var/lib/kubelet/pods/23f4ab2f-43a6-413b-9250-7ad0e808bda7/volumes"
Jul 09 19:33:38.973268 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:38.973235    2556 generic.go:356] "Generic (PLEG): container finished" podID="3bf69fd5-5d2c-4437-891d-57f489765ea5" containerID="a22923f5fa799f95404ab20cdafa0993b2728167b49bb555c40089b5ece62bf7" exitCode=0
Jul 09 19:33:38.973704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:38.973330    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307190338480" event={"ID":"3bf69fd5-5d2c-4437-891d-57f489765ea5","Type":"ContainerDied","Data":"a22923f5fa799f95404ab20cdafa0993b2728167b49bb555c40089b5ece62bf7"}
Jul 09 19:33:40.128154 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.128129    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:40.158519 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.158469    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7482-140307190338480_3bf69fd5-5d2c-4437-891d-57f489765ea5/test-curl-7482-140307190338480/0.log"
Jul 09 19:33:40.206475 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.206441    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7482-140307190338480"]
Jul 09 19:33:40.219174 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.219142    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7482-140307190338480"]
Jul 09 19:33:40.229386 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.229315    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsh5p\" (UniqueName: \"kubernetes.io/projected/3bf69fd5-5d2c-4437-891d-57f489765ea5-kube-api-access-nsh5p\") pod \"3bf69fd5-5d2c-4437-891d-57f489765ea5\" (UID: \"3bf69fd5-5d2c-4437-891d-57f489765ea5\") "
Jul 09 19:33:40.231447 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.231423    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bf69fd5-5d2c-4437-891d-57f489765ea5-kube-api-access-nsh5p" (OuterVolumeSpecName: "kube-api-access-nsh5p") pod "3bf69fd5-5d2c-4437-891d-57f489765ea5" (UID: "3bf69fd5-5d2c-4437-891d-57f489765ea5"). InnerVolumeSpecName "kube-api-access-nsh5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:33:40.330296 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.330257    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsh5p\" (UniqueName: \"kubernetes.io/projected/3bf69fd5-5d2c-4437-891d-57f489765ea5-kube-api-access-nsh5p\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:33:40.403272 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.403232    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:40.403700 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.403686    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3bf69fd5-5d2c-4437-891d-57f489765ea5" containerName="test-curl-7482-140307190338480"
Jul 09 19:33:40.403751 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.403701    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3bf69fd5-5d2c-4437-891d-57f489765ea5" containerName="test-curl-7482-140307190338480"
Jul 09 19:33:40.403791 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.403783    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3bf69fd5-5d2c-4437-891d-57f489765ea5" containerName="test-curl-7482-140307190338480"
Jul 09 19:33:40.406755 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.406737    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:40.426632 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.426595    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:40.532024 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.531933    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8rbf\" (UniqueName: \"kubernetes.io/projected/262110ce-7457-4dce-96c2-d3e07e860387-kube-api-access-c8rbf\") pod \"test-curl-7482-140307192412032\" (UID: \"262110ce-7457-4dce-96c2-d3e07e860387\") " pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:40.633316 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.633280    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8rbf\" (UniqueName: \"kubernetes.io/projected/262110ce-7457-4dce-96c2-d3e07e860387-kube-api-access-c8rbf\") pod \"test-curl-7482-140307192412032\" (UID: \"262110ce-7457-4dce-96c2-d3e07e860387\") " pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:40.643704 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.643678    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8rbf\" (UniqueName: \"kubernetes.io/projected/262110ce-7457-4dce-96c2-d3e07e860387-kube-api-access-c8rbf\") pod \"test-curl-7482-140307192412032\" (UID: \"262110ce-7457-4dce-96c2-d3e07e860387\") " pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:40.716439 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.716401    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:40.870897 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.870864    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:40.873766 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:33:40.873736    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod262110ce_7457_4dce_96c2_d3e07e860387.slice/crio-b3d0ec87e10da1c17d39dff2d0d6cdda990ed1d5cf778502b3be959a8dc878ca WatchSource:0}: Error finding container b3d0ec87e10da1c17d39dff2d0d6cdda990ed1d5cf778502b3be959a8dc878ca: Status 404 returned error can't find the container with id b3d0ec87e10da1c17d39dff2d0d6cdda990ed1d5cf778502b3be959a8dc878ca
Jul 09 19:33:40.985257 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.985224    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192412032" event={"ID":"262110ce-7457-4dce-96c2-d3e07e860387","Type":"ContainerStarted","Data":"b3d0ec87e10da1c17d39dff2d0d6cdda990ed1d5cf778502b3be959a8dc878ca"}
Jul 09 19:33:40.986515 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.986494    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="781b8134c735310d9445b88b0d5d596aa73bbd4bd2add18c450ff1ea9852a415"
Jul 09 19:33:40.986659 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:40.986516    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:40.997342 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:33:40.997314    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307190338480\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="3bf69fd5-5d2c-4437-891d-57f489765ea5" pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:41.993615 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:41.993582    2556 generic.go:356] "Generic (PLEG): container finished" podID="262110ce-7457-4dce-96c2-d3e07e860387" containerID="cf85764469c63cbfe5d7e3b7836921d9c9164a45a7e8f6ee42e4f2663aad0ece" exitCode=0
Jul 09 19:33:41.993615 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:41.993620    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192412032" event={"ID":"262110ce-7457-4dce-96c2-d3e07e860387","Type":"ContainerDied","Data":"cf85764469c63cbfe5d7e3b7836921d9c9164a45a7e8f6ee42e4f2663aad0ece"}
Jul 09 19:33:41.995956 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:33:41.995930    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307190338480\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="3bf69fd5-5d2c-4437-891d-57f489765ea5" pod="openshift-ingress/test-curl-7482-140307190338480"
Jul 09 19:33:42.086522 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:42.086487    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3bf69fd5-5d2c-4437-891d-57f489765ea5" path="/var/lib/kubelet/pods/3bf69fd5-5d2c-4437-891d-57f489765ea5/volumes"
Jul 09 19:33:43.154246 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.154218    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:43.176454 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.176424    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7482-140307192412032_262110ce-7457-4dce-96c2-d3e07e860387/test-curl-7482-140307192412032/0.log"
Jul 09 19:33:43.202847 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.202810    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:43.206766 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.206741    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:43.255604 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.255567    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8rbf\" (UniqueName: \"kubernetes.io/projected/262110ce-7457-4dce-96c2-d3e07e860387-kube-api-access-c8rbf\") pod \"262110ce-7457-4dce-96c2-d3e07e860387\" (UID: \"262110ce-7457-4dce-96c2-d3e07e860387\") "
Jul 09 19:33:43.257791 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.257732    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/262110ce-7457-4dce-96c2-d3e07e860387-kube-api-access-c8rbf" (OuterVolumeSpecName: "kube-api-access-c8rbf") pod "262110ce-7457-4dce-96c2-d3e07e860387" (UID: "262110ce-7457-4dce-96c2-d3e07e860387"). InnerVolumeSpecName "kube-api-access-c8rbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:33:43.328490 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.328458    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:43.328931 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.328917    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="262110ce-7457-4dce-96c2-d3e07e860387" containerName="test-curl-7482-140307192412032"
Jul 09 19:33:43.328975 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.328932    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="262110ce-7457-4dce-96c2-d3e07e860387" containerName="test-curl-7482-140307192412032"
Jul 09 19:33:43.329019 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.329009    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="262110ce-7457-4dce-96c2-d3e07e860387" containerName="test-curl-7482-140307192412032"
Jul 09 19:33:43.332134 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.332118    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:43.343307 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.343271    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:43.357173 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.357143    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8rbf\" (UniqueName: \"kubernetes.io/projected/262110ce-7457-4dce-96c2-d3e07e860387-kube-api-access-c8rbf\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:33:43.458323 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.458282    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kt2dq\" (UniqueName: \"kubernetes.io/projected/b902f374-e660-46ed-9fa7-753facf6ff1a-kube-api-access-kt2dq\") pod \"test-curl-7482-140307192412032\" (UID: \"b902f374-e660-46ed-9fa7-753facf6ff1a\") " pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:43.559513 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.559409    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kt2dq\" (UniqueName: \"kubernetes.io/projected/b902f374-e660-46ed-9fa7-753facf6ff1a-kube-api-access-kt2dq\") pod \"test-curl-7482-140307192412032\" (UID: \"b902f374-e660-46ed-9fa7-753facf6ff1a\") " pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:43.569728 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.569701    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kt2dq\" (UniqueName: \"kubernetes.io/projected/b902f374-e660-46ed-9fa7-753facf6ff1a-kube-api-access-kt2dq\") pod \"test-curl-7482-140307192412032\" (UID: \"b902f374-e660-46ed-9fa7-753facf6ff1a\") " pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:43.641419 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.641380    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:43.795977 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:43.795953    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:43.798202 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:33:43.798175    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb902f374_e660_46ed_9fa7_753facf6ff1a.slice/crio-26ba5603a6786af76ac39e5501a158269308402dab6f1669b581e8f1064a1493 WatchSource:0}: Error finding container 26ba5603a6786af76ac39e5501a158269308402dab6f1669b581e8f1064a1493: Status 404 returned error can't find the container with id 26ba5603a6786af76ac39e5501a158269308402dab6f1669b581e8f1064a1493
Jul 09 19:33:44.005511 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:44.005475    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192412032" event={"ID":"b902f374-e660-46ed-9fa7-753facf6ff1a","Type":"ContainerStarted","Data":"26ba5603a6786af76ac39e5501a158269308402dab6f1669b581e8f1064a1493"}
Jul 09 19:33:44.006905 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:44.006884    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b3d0ec87e10da1c17d39dff2d0d6cdda990ed1d5cf778502b3be959a8dc878ca"
Jul 09 19:33:44.007032 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:44.006910    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:44.019646 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:44.019613    2556 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="openshift-ingress/test-curl-7482-140307192412032" oldPodUID="262110ce-7457-4dce-96c2-d3e07e860387" podUID="b902f374-e660-46ed-9fa7-753facf6ff1a"
Jul 09 19:33:44.085143 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:44.085065    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="262110ce-7457-4dce-96c2-d3e07e860387" path="/var/lib/kubelet/pods/262110ce-7457-4dce-96c2-d3e07e860387/volumes"
Jul 09 19:33:45.013132 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:45.013097    2556 generic.go:356] "Generic (PLEG): container finished" podID="b902f374-e660-46ed-9fa7-753facf6ff1a" containerID="4a91dc4174d53d7ac5ff4a2f2cee32fd588633adb461fb7c2bdcd1c77e01b44a" exitCode=0
Jul 09 19:33:45.013502 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:45.013180    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192412032" event={"ID":"b902f374-e660-46ed-9fa7-753facf6ff1a","Type":"ContainerDied","Data":"4a91dc4174d53d7ac5ff4a2f2cee32fd588633adb461fb7c2bdcd1c77e01b44a"}
Jul 09 19:33:46.188127 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.188099    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:46.213270 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.213244    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7482-140307192412032_b902f374-e660-46ed-9fa7-753facf6ff1a/test-curl-7482-140307192412032/0.log"
Jul 09 19:33:46.243958 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.243925    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:46.247462 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.247439    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7482-140307192412032"]
Jul 09 19:33:46.284601 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.284516    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kt2dq\" (UniqueName: \"kubernetes.io/projected/b902f374-e660-46ed-9fa7-753facf6ff1a-kube-api-access-kt2dq\") pod \"b902f374-e660-46ed-9fa7-753facf6ff1a\" (UID: \"b902f374-e660-46ed-9fa7-753facf6ff1a\") "
Jul 09 19:33:46.286557 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.286516    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b902f374-e660-46ed-9fa7-753facf6ff1a-kube-api-access-kt2dq" (OuterVolumeSpecName: "kube-api-access-kt2dq") pod "b902f374-e660-46ed-9fa7-753facf6ff1a" (UID: "b902f374-e660-46ed-9fa7-753facf6ff1a"). InnerVolumeSpecName "kube-api-access-kt2dq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:33:46.370438 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.370400    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"]
Jul 09 19:33:46.370923 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.370905    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b902f374-e660-46ed-9fa7-753facf6ff1a" containerName="test-curl-7482-140307192412032"
Jul 09 19:33:46.371035 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.370925    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="b902f374-e660-46ed-9fa7-753facf6ff1a" containerName="test-curl-7482-140307192412032"
Jul 09 19:33:46.371097 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.371033    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="b902f374-e660-46ed-9fa7-753facf6ff1a" containerName="test-curl-7482-140307192412032"
Jul 09 19:33:46.374115 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.374095    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:46.385809 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.385786    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kt2dq\" (UniqueName: \"kubernetes.io/projected/b902f374-e660-46ed-9fa7-753facf6ff1a-kube-api-access-kt2dq\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:33:46.388642 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.388616    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"]
Jul 09 19:33:46.486864 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.486826    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj2vn\" (UniqueName: \"kubernetes.io/projected/52bee9ed-8043-425c-92bf-497de04b4540-kube-api-access-vj2vn\") pod \"test-curl-7482-140307192413040\" (UID: \"52bee9ed-8043-425c-92bf-497de04b4540\") " pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:46.587837 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.587740    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj2vn\" (UniqueName: \"kubernetes.io/projected/52bee9ed-8043-425c-92bf-497de04b4540-kube-api-access-vj2vn\") pod \"test-curl-7482-140307192413040\" (UID: \"52bee9ed-8043-425c-92bf-497de04b4540\") " pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:46.597190 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.597163    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj2vn\" (UniqueName: \"kubernetes.io/projected/52bee9ed-8043-425c-92bf-497de04b4540-kube-api-access-vj2vn\") pod \"test-curl-7482-140307192413040\" (UID: \"52bee9ed-8043-425c-92bf-497de04b4540\") " pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:46.683430 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.683394    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:46.822485 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:46.822462    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"]
Jul 09 19:33:46.824479 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:33:46.824446    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52bee9ed_8043_425c_92bf_497de04b4540.slice/crio-d2463c340888d9d3721af17b60db42025096e251066fe70baadf24b319fe96fb WatchSource:0}: Error finding container d2463c340888d9d3721af17b60db42025096e251066fe70baadf24b319fe96fb: Status 404 returned error can't find the container with id d2463c340888d9d3721af17b60db42025096e251066fe70baadf24b319fe96fb
Jul 09 19:33:47.025414 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:47.025381    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192413040" event={"ID":"52bee9ed-8043-425c-92bf-497de04b4540","Type":"ContainerStarted","Data":"d2463c340888d9d3721af17b60db42025096e251066fe70baadf24b319fe96fb"}
Jul 09 19:33:47.026606 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:47.026588    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26ba5603a6786af76ac39e5501a158269308402dab6f1669b581e8f1064a1493"
Jul 09 19:33:47.026726 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:47.026631    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:47.037176 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:33:47.037151    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307192412032\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="b902f374-e660-46ed-9fa7-753facf6ff1a" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:48.033133 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:48.033103    2556 generic.go:356] "Generic (PLEG): container finished" podID="52bee9ed-8043-425c-92bf-497de04b4540" containerID="f4b9311a552b9e0c256fe2d393f8c07fc751f83442273fc12db703d435891ee8" exitCode=0
Jul 09 19:33:48.033562 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:48.033181    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7482-140307192413040" event={"ID":"52bee9ed-8043-425c-92bf-497de04b4540","Type":"ContainerDied","Data":"f4b9311a552b9e0c256fe2d393f8c07fc751f83442273fc12db703d435891ee8"}
Jul 09 19:33:48.035848 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:33:48.035825    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307192412032\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="b902f374-e660-46ed-9fa7-753facf6ff1a" pod="openshift-ingress/test-curl-7482-140307192412032"
Jul 09 19:33:48.085336 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:48.085302    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b902f374-e660-46ed-9fa7-753facf6ff1a" path="/var/lib/kubelet/pods/b902f374-e660-46ed-9fa7-753facf6ff1a/volumes"
Jul 09 19:33:49.202596 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:49.202570    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:49.212102 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:49.212081    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vj2vn\" (UniqueName: \"kubernetes.io/projected/52bee9ed-8043-425c-92bf-497de04b4540-kube-api-access-vj2vn\") pod \"52bee9ed-8043-425c-92bf-497de04b4540\" (UID: \"52bee9ed-8043-425c-92bf-497de04b4540\") "
Jul 09 19:33:49.214065 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:49.214035    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52bee9ed-8043-425c-92bf-497de04b4540-kube-api-access-vj2vn" (OuterVolumeSpecName: "kube-api-access-vj2vn") pod "52bee9ed-8043-425c-92bf-497de04b4540" (UID: "52bee9ed-8043-425c-92bf-497de04b4540"). InnerVolumeSpecName "kube-api-access-vj2vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:33:49.225671 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:49.225645    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7482-140307192413040_52bee9ed-8043-425c-92bf-497de04b4540/test-curl-7482-140307192413040/0.log"
Jul 09 19:33:49.252258 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:49.252220    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"]
Jul 09 19:33:49.256008 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:49.255983    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7482-140307192413040"]
Jul 09 19:33:49.312831 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:49.312739    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj2vn\" (UniqueName: \"kubernetes.io/projected/52bee9ed-8043-425c-92bf-497de04b4540-kube-api-access-vj2vn\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:33:50.044864 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:50.044836    2556 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d2463c340888d9d3721af17b60db42025096e251066fe70baadf24b319fe96fb"
Jul 09 19:33:50.044864 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:50.044838    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:50.055334 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:33:50.055305    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307192413040\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="52bee9ed-8043-425c-92bf-497de04b4540" pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:50.084318 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:33:50.084284    2556 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7482-140307192413040\" is forbidden: User \"system:node:ip-10-0-129-93.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-93.ec2.internal' and this object" podUID="52bee9ed-8043-425c-92bf-497de04b4540" pod="openshift-ingress/test-curl-7482-140307192413040"
Jul 09 19:33:50.085684 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:50.085662    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="52bee9ed-8043-425c-92bf-497de04b4540" path="/var/lib/kubelet/pods/52bee9ed-8043-425c-92bf-497de04b4540/volumes"
Jul 09 19:33:53.066011 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:53.065976    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:53.469407 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:53.469370    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:53.568146 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:53.568107    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:53.667964 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:53.667930    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:53.770595 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:53.770504    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:59.788846 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:59.788811    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:33:59.923874 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:33:59.923841    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:00.029540 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:00.029507    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:00.241689 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:00.241648    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:00.410378 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:00.410346    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:00.819154 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:00.819122    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:00.962128 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:00.962098    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:01.065784 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.065755    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:01.226600 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.226568    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"]
Jul 09 19:34:01.226881 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.226850    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn" podUID="3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" containerName="maas-api" containerID="cri-o://f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39" gracePeriod=30
Jul 09 19:34:01.253821 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.253792    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:01.484550 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.484478    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:34:01.520429 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.520399    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls\") pod \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") "
Jul 09 19:34:01.520603 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.520455    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-openshift-service-ca\") pod \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") "
Jul 09 19:34:01.520603 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.520479    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kjhjx\" (UniqueName: \"kubernetes.io/projected/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-kube-api-access-kjhjx\") pod \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\" (UID: \"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7\") "
Jul 09 19:34:01.520848 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.520823    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" (UID: "3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:34:01.522654 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.522620    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" (UID: "3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 09 19:34:01.522654 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.522636    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-kube-api-access-kjhjx" (OuterVolumeSpecName: "kube-api-access-kjhjx") pod "3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" (UID: "3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7"). InnerVolumeSpecName "kube-api-access-kjhjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:34:01.621976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.621937    2556 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-openshift-service-ca\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:01.621976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.621969    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjhjx\" (UniqueName: \"kubernetes.io/projected/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-kube-api-access-kjhjx\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:01.621976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:01.621982    2556 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7-maas-api-tls\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:02.113254 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.113222    2556 generic.go:356] "Generic (PLEG): container finished" podID="3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" containerID="f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39" exitCode=0
Jul 09 19:34:02.113720 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.113280    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"
Jul 09 19:34:02.113720 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.113289    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn" event={"ID":"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7","Type":"ContainerDied","Data":"f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39"}
Jul 09 19:34:02.113720 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.113343    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn" event={"ID":"3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7","Type":"ContainerDied","Data":"685559a39de4afb9b82596800a4989d669d1a1a0626de448e693959bd9d9150d"}
Jul 09 19:34:02.113720 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.113370    2556 scope.go:117] "RemoveContainer" containerID="f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39"
Jul 09 19:34:02.122301 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.122276    2556 scope.go:117] "RemoveContainer" containerID="f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39"
Jul 09 19:34:02.122545 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:34:02.122509    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39\": container with ID starting with f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39 not found: ID does not exist" containerID="f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39"
Jul 09 19:34:02.122605 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.122546    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39"} err="failed to get container status \"f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39\": rpc error: code = NotFound desc = could not find container \"f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39\": container with ID starting with f25fb715df3e6e1b6faac7ba702a69e3e3751d44fa4e160694388626ae6d6b39 not found: ID does not exist"
Jul 09 19:34:02.131106 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.131080    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"]
Jul 09 19:34:02.134982 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:02.134962    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-b20a51-84b956dbff-vw8dn"]
Jul 09 19:34:04.086081 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:04.086049    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" path="/var/lib/kubelet/pods/3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7/volumes"
Jul 09 19:34:06.816464 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:06.816425    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:06.916614 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:06.916586    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:07.076225 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:07.076123    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"]
Jul 09 19:34:07.076416 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:07.076384    2556 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy" containerID="cri-o://2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d" gracePeriod=30
Jul 09 19:34:07.085844 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:07.085817    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:07.325094 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:07.325060    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:12.343600 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.343508    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:34:12.423463 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423426    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-credential-socket\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.423463 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423470    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-token\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.423746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423490    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-data\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.423746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423507    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-podinfo\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.423746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423566    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-certs\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.423746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423596    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-socket\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.423746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423632    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istiod-ca-cert\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.423746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423672    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-envoy\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.424044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423768    2556 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5tmms\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-kube-api-access-5tmms\") pod \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\" (UID: \"280a1a95-e8b1-4c52-822f-8c4a07c900bb\") "
Jul 09 19:34:12.424044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423939    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:34:12.424044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.423961    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:34:12.424044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.424027    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:34:12.424230 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.424104    2556 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-credential-socket\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.424230 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.424124    2556 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-certs\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.424349 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.424325    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 09 19:34:12.424683 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.424659    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-data" (OuterVolumeSpecName: "istio-data") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:34:12.425885 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.425865    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-token" (OuterVolumeSpecName: "istio-token") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:34:12.426053 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.426029    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 09 19:34:12.426217 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.426192    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-kube-api-access-5tmms" (OuterVolumeSpecName: "kube-api-access-5tmms") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "kube-api-access-5tmms". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 09 19:34:12.426314 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.426187    2556 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "280a1a95-e8b1-4c52-822f-8c4a07c900bb" (UID: "280a1a95-e8b1-4c52-822f-8c4a07c900bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 09 19:34:12.524787 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.524742    2556 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5tmms\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-kube-api-access-5tmms\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.524787 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.524780    2556 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-token\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.524787 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.524795    2556 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-data\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.525044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.524808    2556 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-podinfo\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.525044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.524820    2556 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-workload-socket\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.525044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.524830    2556 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istiod-ca-cert\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.525044 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.524841    2556 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a1a95-e8b1-4c52-822f-8c4a07c900bb-istio-envoy\") on node \"ip-10-0-129-93.ec2.internal\" DevicePath \"\""
Jul 09 19:34:12.707128 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.707093    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-57w4c_d6d2f532-9346-4908-a5d8-3a06cea409b0/manager/0.log"
Jul 09 19:34:12.956662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:12.956628    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5495d599d9-ht6jg_2791912b-a71c-4636-8269-ca88a46628da/manager/0.log"
Jul 09 19:34:13.079542 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.079448    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p8rb6_2aeba00d-5159-4b98-ad8c-dc649da0f565/server/0.log"
Jul 09 19:34:13.175881 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.175844    2556 generic.go:356] "Generic (PLEG): container finished" podID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerID="2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d" exitCode=0
Jul 09 19:34:13.176075 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.175923    2556 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"
Jul 09 19:34:13.176075 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.175974    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" event={"ID":"280a1a95-e8b1-4c52-822f-8c4a07c900bb","Type":"ContainerDied","Data":"2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d"}
Jul 09 19:34:13.176075 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.176054    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2" event={"ID":"280a1a95-e8b1-4c52-822f-8c4a07c900bb","Type":"ContainerDied","Data":"a7efb900860e13177eaf5cc31d37115064a9d3323d6bbfc0713c5c0a70a66e31"}
Jul 09 19:34:13.176206 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.176091    2556 scope.go:117] "RemoveContainer" containerID="2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d"
Jul 09 19:34:13.186231 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.186213    2556 scope.go:117] "RemoveContainer" containerID="823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99"
Jul 09 19:34:13.196555 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.196347    2556 scope.go:117] "RemoveContainer" containerID="2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d"
Jul 09 19:34:13.196729 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:34:13.196702    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d\": container with ID starting with 2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d not found: ID does not exist" containerID="2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d"
Jul 09 19:34:13.196795 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.196744    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d"} err="failed to get container status \"2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d\": rpc error: code = NotFound desc = could not find container \"2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d\": container with ID starting with 2825aef7ebf43087aab52b723e4fa94461d56da1c0f873a48cc1e2d866942c5d not found: ID does not exist"
Jul 09 19:34:13.196795 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.196764    2556 scope.go:117] "RemoveContainer" containerID="823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99"
Jul 09 19:34:13.197100 ip-10-0-129-93 kubenswrapper[2556]: E0709 19:34:13.197063    2556 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99\": container with ID starting with 823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99 not found: ID does not exist" containerID="823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99"
Jul 09 19:34:13.197216 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.197109    2556 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99"} err="failed to get container status \"823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99\": rpc error: code = NotFound desc = could not find container \"823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99\": container with ID starting with 823889f57f839056fcf2b7e7ab06333dab7affe2166a2a93f6bc7d0daed04b99 not found: ID does not exist"
Jul 09 19:34:13.198314 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.198290    2556 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"]
Jul 09 19:34:13.198609 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.198591    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j97g2_885633a5-f210-4edc-b348-df292898fc5c/manager/0.log"
Jul 09 19:34:13.205927 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.205906    2556 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-b20a51-openshift-default-66cc8b8f79-pn9m2"]
Jul 09 19:34:13.313067 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.313035    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-ddcnl_e24ffc53-a74f-4ee1-b917-bff6147474f2/manager/0.log"
Jul 09 19:34:13.319939 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.319920    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-ddcnl_e24ffc53-a74f-4ee1-b917-bff6147474f2/copy-manifests/0.log"
Jul 09 19:34:13.941941 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:13.941910    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5d69b8c867-v968d_d92bbf26-9bab-451c-b5b0-65262d2ffbca/maas-api/0.log"
Jul 09 19:34:14.067229 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:14.067199    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-4s48q_dcbf8e6e-c9ab-4ca6-8b32-f631a9c1e8ff/postgres/0.log"
Jul 09 19:34:14.085410 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:14.085371    2556 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" path="/var/lib/kubelet/pods/280a1a95-e8b1-4c52-822f-8c4a07c900bb/volumes"
Jul 09 19:34:15.559350 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:15.559316    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9w2mn_34a6488d-1d4c-446e-a390-cbc98c38b25c/manager/0.log"
Jul 09 19:34:15.676194 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:15.676170    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m4shg_4d3243d2-2bf7-431c-a3a1-5f40a1325f08/manager/0.log"
Jul 09 19:34:15.911625 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:15.911541    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fhjm2_fb30b66d-1c28-47e0-aeb8-257d0852e646/registry-server/0.log"
Jul 09 19:34:16.030598 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:16.030558    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-c694q_c337e848-7ba3-423e-9bc2-80b3fc391ae1/manager/0.log"
Jul 09 19:34:16.274319 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:16.274293    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-brm9r_608b40b5-6d2b-41ea-a49e-489e44f78eab/manager/0.log"
Jul 09 19:34:16.988169 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:16.988140    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-zg657_61ab7487-5540-4a64-aa38-ed6e7a449d5c/kube-auth-proxy/0.log"
Jul 09 19:34:17.361654 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:17.361574    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vzzxw_150574cd-7333-49b7-b727-782f54561744/payload-processing/0.log"
Jul 09 19:34:17.487384 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:17.487361    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66ddf58c96-f6w75_7d5933ef-2952-4201-891b-78f29352a829/router/0.log"
Jul 09 19:34:17.953032 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:17.953000    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-2xmnr_e4d9e3c4-5595-4828-8374-154e6bcf2a1c/main/0.log"
Jul 09 19:34:22.910976 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.910938    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fz6n6/must-gather-98ckm"]
Jul 09 19:34:22.911370 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911362    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="52bee9ed-8043-425c-92bf-497de04b4540" containerName="test-curl-7482-140307192413040"
Jul 09 19:34:22.911413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911374    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="52bee9ed-8043-425c-92bf-497de04b4540" containerName="test-curl-7482-140307192413040"
Jul 09 19:34:22.911413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911386    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" containerName="maas-api"
Jul 09 19:34:22.911413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911392    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" containerName="maas-api"
Jul 09 19:34:22.911413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911406    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy"
Jul 09 19:34:22.911413 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911412    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy"
Jul 09 19:34:22.911597 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911419    2556 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy"
Jul 09 19:34:22.911597 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911425    2556 state_mem.go:107] "Deleted CPUSet assignment" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy"
Jul 09 19:34:22.911597 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911495    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy"
Jul 09 19:34:22.911597 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911502    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="3c3dd1bf-d46a-48c9-a293-eef8a24f4cb7" containerName="maas-api"
Jul 09 19:34:22.911597 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911508    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="280a1a95-e8b1-4c52-822f-8c4a07c900bb" containerName="istio-proxy"
Jul 09 19:34:22.911597 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.911516    2556 memory_manager.go:368] "RemoveStaleState removing state" podUID="52bee9ed-8043-425c-92bf-497de04b4540" containerName="test-curl-7482-140307192413040"
Jul 09 19:34:22.913793 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.913772    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:22.917584 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.917543    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"openshift-service-ca.crt\""
Jul 09 19:34:22.917734 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.917714    2556 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-fz6n6\"/\"default-dockercfg-pl4vb\""
Jul 09 19:34:22.918777 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.918755    2556 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-fz6n6\"/\"kube-root-ca.crt\""
Jul 09 19:34:22.919793 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.919765    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:22.924783 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:22.924754    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/must-gather-98ckm"]
Jul 09 19:34:23.015920 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.015887    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:23.023910 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.023879    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86e5f85e-1ee3-46fb-8991-99250e1446d2-must-gather-output\") pod \"must-gather-98ckm\" (UID: \"86e5f85e-1ee3-46fb-8991-99250e1446d2\") " pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:23.024048 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.023942    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xkvgh\" (UniqueName: \"kubernetes.io/projected/86e5f85e-1ee3-46fb-8991-99250e1446d2-kube-api-access-xkvgh\") pod \"must-gather-98ckm\" (UID: \"86e5f85e-1ee3-46fb-8991-99250e1446d2\") " pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:23.124860 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.124822    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xkvgh\" (UniqueName: \"kubernetes.io/projected/86e5f85e-1ee3-46fb-8991-99250e1446d2-kube-api-access-xkvgh\") pod \"must-gather-98ckm\" (UID: \"86e5f85e-1ee3-46fb-8991-99250e1446d2\") " pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:23.125048 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.124900    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86e5f85e-1ee3-46fb-8991-99250e1446d2-must-gather-output\") pod \"must-gather-98ckm\" (UID: \"86e5f85e-1ee3-46fb-8991-99250e1446d2\") " pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:23.125226 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.125209    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/86e5f85e-1ee3-46fb-8991-99250e1446d2-must-gather-output\") pod \"must-gather-98ckm\" (UID: \"86e5f85e-1ee3-46fb-8991-99250e1446d2\") " pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:23.133536 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.133507    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xkvgh\" (UniqueName: \"kubernetes.io/projected/86e5f85e-1ee3-46fb-8991-99250e1446d2-kube-api-access-xkvgh\") pod \"must-gather-98ckm\" (UID: \"86e5f85e-1ee3-46fb-8991-99250e1446d2\") " pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:23.224085 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.224049    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/must-gather-98ckm"
Jul 09 19:34:23.376388 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:23.376359    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/must-gather-98ckm"]
Jul 09 19:34:23.378582 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:34:23.378553    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod86e5f85e_1ee3_46fb_8991_99250e1446d2.slice/crio-895cfb37591f544cfab245baa453a93be7f9c760b1bb59d7322039decbf0406b WatchSource:0}: Error finding container 895cfb37591f544cfab245baa453a93be7f9c760b1bb59d7322039decbf0406b: Status 404 returned error can't find the container with id 895cfb37591f544cfab245baa453a93be7f9c760b1bb59d7322039decbf0406b
Jul 09 19:34:24.244490 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:24.244452    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/must-gather-98ckm" event={"ID":"86e5f85e-1ee3-46fb-8991-99250e1446d2","Type":"ContainerStarted","Data":"11c735c4fefc905becc83b11e2bb374527a584694d53d01c185f1060cc83e4ce"}
Jul 09 19:34:24.245126 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:24.245043    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/must-gather-98ckm" event={"ID":"86e5f85e-1ee3-46fb-8991-99250e1446d2","Type":"ContainerStarted","Data":"9443ee95f84417e26d0b3ee145b18b7eed27cd60459c144a873e547df7d84b62"}
Jul 09 19:34:24.245126 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:24.245086    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/must-gather-98ckm" event={"ID":"86e5f85e-1ee3-46fb-8991-99250e1446d2","Type":"ContainerStarted","Data":"895cfb37591f544cfab245baa453a93be7f9c760b1bb59d7322039decbf0406b"}
Jul 09 19:34:24.265320 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:24.265271    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fz6n6/must-gather-98ckm" podStartSLOduration=1.620999693 podStartE2EDuration="2.265255432s" totalImagesPullingTime="644.255739ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-09 19:34:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:34:24.262004114 +0000 UTC m=+2474.614439181" watchObservedRunningTime="2026-07-09 19:34:24.265255432 +0000 UTC m=+2474.617690498"
Jul 09 19:34:25.736958 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:25.736925    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-v9667_f721565e-88df-4cad-b2e4-49193dcfbddd/global-pull-secret-syncer/0.log"
Jul 09 19:34:25.800032 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:25.799995    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log"
Jul 09 19:34:25.919385 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:25.919355    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-93.ec2.internal_50ac5346786c8f7d0f6951a8eaeafe6e/haproxy/0.log"
Jul 09 19:34:32.072821 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:32.072759    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-9w2mn_34a6488d-1d4c-446e-a390-cbc98c38b25c/manager/0.log"
Jul 09 19:34:32.102713 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:32.102684    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-m4shg_4d3243d2-2bf7-431c-a3a1-5f40a1325f08/manager/0.log"
Jul 09 19:34:32.185299 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:32.185259    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-fhjm2_fb30b66d-1c28-47e0-aeb8-257d0852e646/registry-server/0.log"
Jul 09 19:34:32.309241 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:32.309205    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-c694q_c337e848-7ba3-423e-9bc2-80b3fc391ae1/manager/0.log"
Jul 09 19:34:32.421545 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:32.421248    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-859d7b78bd-brm9r_608b40b5-6d2b-41ea-a49e-489e44f78eab/manager/0.log"
Jul 09 19:34:34.204920 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:34.204887    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_cluster-monitoring-operator-5f7f494dd4-rbkwz_bc39ed10-3fcd-47e9-98af-2940484eb9ca/cluster-monitoring-operator/0.log"
Jul 09 19:34:34.523603 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:34.523498    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95qtj_85eb96a2-9d58-4ff3-b18e-58e30877ff63/node-exporter/0.log"
Jul 09 19:34:34.556030 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:34.556003    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95qtj_85eb96a2-9d58-4ff3-b18e-58e30877ff63/kube-rbac-proxy/0.log"
Jul 09 19:34:34.579460 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:34.579359    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-95qtj_85eb96a2-9d58-4ff3-b18e-58e30877ff63/init-textfile/0.log"
Jul 09 19:34:34.859207 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:34.859117    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-rsgtq_7234fe0c-4489-41b3-8868-7de2232675f7/prometheus-operator/0.log"
Jul 09 19:34:34.880293 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:34.880224    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-rsgtq_7234fe0c-4489-41b3-8868-7de2232675f7/kube-rbac-proxy/0.log"
Jul 09 19:34:34.908007 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:34.907980    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-zxx77_407c46f4-31ca-4f90-a108-a2deab7ad827/prometheus-operator-admission-webhook/0.log"
Jul 09 19:34:35.404266 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.404224    2556 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"]
Jul 09 19:34:35.412942 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.412912    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.416501 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.416464    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"]
Jul 09 19:34:35.555210 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.555164    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-lib-modules\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.555517 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.555498    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-podres\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.555746 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.555722    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-sys\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.555892 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.555865    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8psz9\" (UniqueName: \"kubernetes.io/projected/f347b050-df2c-4b41-9f81-2fa018f29e1d-kube-api-access-8psz9\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.555961 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.555916    2556 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-proc\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.656738 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.656618    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-sys\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.656738 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.656726    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8psz9\" (UniqueName: \"kubernetes.io/projected/f347b050-df2c-4b41-9f81-2fa018f29e1d-kube-api-access-8psz9\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.656987 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.656753    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-proc\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.656987 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.656799    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-sys\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.656987 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.656824    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-lib-modules\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.656987 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.656903    2556 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-podres\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.656987 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.656955    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-lib-modules\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.657246 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.657043    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-podres\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.657246 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.657046    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/f347b050-df2c-4b41-9f81-2fa018f29e1d-proc\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.665359 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.665331    2556 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8psz9\" (UniqueName: \"kubernetes.io/projected/f347b050-df2c-4b41-9f81-2fa018f29e1d-kube-api-access-8psz9\") pod \"perf-node-gather-daemonset-2vsd6\" (UID: \"f347b050-df2c-4b41-9f81-2fa018f29e1d\") " pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.727716 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.727678    2556 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:35.932164 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:35.931718    2556 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"]
Jul 09 19:34:35.934338 ip-10-0-129-93 kubenswrapper[2556]: W0709 19:34:35.934307    2556 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podf347b050_df2c_4b41_9f81_2fa018f29e1d.slice/crio-2bf9fa45c8700107832014f72d154d1ff2c65dfb1b86a598c11d97289aa74405 WatchSource:0}: Error finding container 2bf9fa45c8700107832014f72d154d1ff2c65dfb1b86a598c11d97289aa74405: Status 404 returned error can't find the container with id 2bf9fa45c8700107832014f72d154d1ff2c65dfb1b86a598c11d97289aa74405
Jul 09 19:34:36.351879 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:36.351835    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6" event={"ID":"f347b050-df2c-4b41-9f81-2fa018f29e1d","Type":"ContainerStarted","Data":"83bd9649e71081ad43edf3b61031b6a0cff98dee9aebfd6f2999789ae6b74115"}
Jul 09 19:34:36.352078 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:36.351908    2556 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6" event={"ID":"f347b050-df2c-4b41-9f81-2fa018f29e1d","Type":"ContainerStarted","Data":"2bf9fa45c8700107832014f72d154d1ff2c65dfb1b86a598c11d97289aa74405"}
Jul 09 19:34:36.352658 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:36.352629    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:36.421797 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:36.421770    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6cf856758c-t2zpb_0208420e-d48a-418d-b036-9a1a611ad6bf/networking-console-plugin/0.log"
Jul 09 19:34:36.993244 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:36.993186    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/1.log"
Jul 09 19:34:36.999442 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:36.999416    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console-operator_console-operator-7866497494-9j4h9_6fd1740a-a8b9-4cdd-87be-0fa546ddf5e3/console-operator/2.log"
Jul 09 19:34:37.532838 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:37.532812    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-6fb5bb4d87-htgxd_18944fe4-9b38-4ca1-a9bf-be2e301da711/console/0.log"
Jul 09 19:34:37.599352 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:37.599321    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-68c9cbb497-npb4s_ada802fe-ca8b-4f1d-a8aa-920a67908065/download-server/0.log"
Jul 09 19:34:38.200444 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:38.200418    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-lh884_ca773c32-ad49-44eb-855d-64f51ae05dfa/volume-data-source-validator/0.log"
Jul 09 19:34:39.277012 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:39.276987    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5knnr_2000c44b-544b-4072-8d0a-604d4b1e1d47/dns/0.log"
Jul 09 19:34:39.300457 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:39.300434    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-5knnr_2000c44b-544b-4072-8d0a-604d4b1e1d47/kube-rbac-proxy/0.log"
Jul 09 19:34:39.440819 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:39.440769    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-jkf4m_58622854-61de-4acb-a671-1cd2aa16fc97/dns-node-resolver/0.log"
Jul 09 19:34:40.052519 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:40.052480    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9qhwv_b67611f8-3dcd-4a7f-8948-3ea4b324e439/node-ca/0.log"
Jul 09 19:34:41.340446 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:41.340416    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-zg657_61ab7487-5540-4a64-aa38-ed6e7a449d5c/kube-auth-proxy/0.log"
Jul 09 19:34:41.474711 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:41.474642    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vzzxw_150574cd-7333-49b7-b727-782f54561744/payload-processing/0.log"
Jul 09 19:34:41.497644 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:41.497613    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66ddf58c96-f6w75_7d5933ef-2952-4201-891b-78f29352a829/router/0.log"
Jul 09 19:34:42.197949 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:42.197918    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-ltkwk_b49c8ad2-d3d4-44bd-91d1-d454284d0c64/serve-healthcheck-canary/0.log"
Jul 09 19:34:42.745116 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:42.745085    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-7zc7x_569b9f3d-0fa3-425a-9815-a374ea2780a6/insights-operator/0.log"
Jul 09 19:34:42.746263 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:42.746244    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-operator-68d9dd94cf-7zc7x_569b9f3d-0fa3-425a-9815-a374ea2780a6/insights-operator/1.log"
Jul 09 19:34:42.907447 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:42.907416    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sq65v_c0e38406-9aca-45ae-9792-e191317edbf3/kube-rbac-proxy/0.log"
Jul 09 19:34:42.929799 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:42.929772    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sq65v_c0e38406-9aca-45ae-9792-e191317edbf3/exporter/0.log"
Jul 09 19:34:42.954631 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:42.954596    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-sq65v_c0e38406-9aca-45ae-9792-e191317edbf3/extractor/0.log"
Jul 09 19:34:43.380918 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:43.380882    2556 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6"
Jul 09 19:34:43.397235 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:43.397184    2556 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-fz6n6/perf-node-gather-daemonset-2vsd6" podStartSLOduration=8.397168064 podStartE2EDuration="8.397168064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-09 19:34:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-09 19:34:36.37266102 +0000 UTC m=+2486.725096085" watchObservedRunningTime="2026-07-09 19:34:43.397168064 +0000 UTC m=+2493.749603128"
Jul 09 19:34:43.629878 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:43.629836    2556 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-9gvq9_48ed19f4-ab27-4e2b-8a1d-b8bba60c2c0e/konnectivity-agent/0.log" limit=1024
Jul 09 19:34:45.463662 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:45.463627    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-57w4c_d6d2f532-9346-4908-a5d8-3a06cea409b0/manager/0.log"
Jul 09 19:34:45.828158 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:45.828072    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-5495d599d9-ht6jg_2791912b-a71c-4636-8269-ca88a46628da/manager/0.log"
Jul 09 19:34:45.855472 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:45.855444    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-p8rb6_2aeba00d-5159-4b98-ad8c-dc649da0f565/server/0.log"
Jul 09 19:34:45.933351 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:45.933323    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-j97g2_885633a5-f210-4edc-b348-df292898fc5c/manager/0.log"
Jul 09 19:34:45.975050 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:45.975016    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-ddcnl_e24ffc53-a74f-4ee1-b917-bff6147474f2/manager/0.log"
Jul 09 19:34:45.998448 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:45.998420    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5cd98cb899-ddcnl_e24ffc53-a74f-4ee1-b917-bff6147474f2/copy-manifests/0.log"
Jul 09 19:34:46.326262 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:46.326230    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-zg657_61ab7487-5540-4a64-aa38-ed6e7a449d5c/kube-auth-proxy/0.log"
Jul 09 19:34:46.443567 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:46.443506    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vzzxw_150574cd-7333-49b7-b727-782f54561744/payload-processing/0.log"
Jul 09 19:34:46.462058 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:46.462012    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66ddf58c96-f6w75_7d5933ef-2952-4201-891b-78f29352a829/router/0.log"
Jul 09 19:34:47.869355 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:47.869272    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5b58cdb447-zg657_61ab7487-5540-4a64-aa38-ed6e7a449d5c/kube-auth-proxy/0.log"
Jul 09 19:34:47.984440 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:47.984403    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-58f86c7798-vzzxw_150574cd-7333-49b7-b727-782f54561744/payload-processing/0.log"
Jul 09 19:34:47.999667 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:47.999638    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_router-default-66ddf58c96-f6w75_7d5933ef-2952-4201-891b-78f29352a829/router/0.log"
Jul 09 19:34:52.381996 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:52.381965    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xg7mj_849a8bde-788c-449a-bdc3-a7af39d97219/migrator/0.log"
Jul 09 19:34:52.403465 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:52.403437    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-xg7mj_849a8bde-788c-449a-bdc3-a7af39d97219/graceful-termination/0.log"
Jul 09 19:34:52.861485 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:52.861452    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-rgrvz_ff7c7467-a4b1-44aa-8dd8-50ed702d7f87/kube-storage-version-migrator-operator/1.log"
Jul 09 19:34:52.862348 ip-10-0-129-93 kubenswrapper[2556]: I0709 19:34:52.862328    2556 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator-operator_kube-storage-version-migrator-operator-799cb7f958-rgrvz_ff7c7467-a4b1-44aa-8dd8-50ed702d7f87/kube-storage-version-migrator-operator/0.log"