Jul 14 00:44:09.905888 ip-10-0-129-125 systemd[1]: Starting Kubernetes Kubelet...
Jul 14 00:44:10.284741 ip-10-0-129-125 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:44:10.284741 ip-10-0-129-125 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 14 00:44:10.284741 ip-10-0-129-125 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:44:10.284741 ip-10-0-129-125 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 14 00:44:10.284741 ip-10-0-129-125 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 14 00:44:10.285516 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.285296    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 14 00:44:10.290782 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290761    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:44:10.290782 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290779    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:44:10.290782 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290784    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:44:10.290782 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290788    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290793    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290797    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290800    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290803    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290806    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290809    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290812    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290817    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290822    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290825    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290829    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290832    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290835    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290838    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290842    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290852    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290856    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290859    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:44:10.290940 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290862    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290865    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290868    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290871    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290874    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290877    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290880    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290883    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290886    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290889    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290891    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290894    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290896    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290899    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290925    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290930    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290934    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290938    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290941    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290945    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:44:10.291416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290949    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290952    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290957    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290962    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290965    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290968    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290970    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290973    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290976    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290979    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290982    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290984    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290987    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290990    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290993    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290996    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.290999    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291002    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291004    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291007    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:44:10.291983 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291009    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291012    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291015    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291017    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291020    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291022    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291025    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291028    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291031    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291036    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291039    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291041    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291044    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291047    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291050    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291052    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291057    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291060    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291063    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291066    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:44:10.292495 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291069    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291072    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291074    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291078    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291081    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291083    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291086    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291089    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291091    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291094    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291097    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291099    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291102    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291104    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291107    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291110    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291112    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291902    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291911    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291915    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:44:10.293009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291919    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291922    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291927    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291931    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291933    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291937    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291940    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291943    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291946    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291948    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291952    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291954    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291957    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291960    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291963    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291966    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291969    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291971    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291974    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:44:10.293515 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291977    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291980    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291983    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291986    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291989    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291992    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291994    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.291997    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292000    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292003    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292005    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292008    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292012    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292015    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292018    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292021    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292024    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292027    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292030    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292032    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:44:10.294020 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292035    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292038    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292040    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292043    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292046    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292049    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292051    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292054    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292057    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292059    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292062    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292064    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292067    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292070    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292072    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292075    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292079    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292082    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292084    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292087    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:44:10.294554 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292090    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292094    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292098    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292101    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292105    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292108    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292112    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292114    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292117    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292120    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292123    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292126    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292129    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292131    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292134    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292138    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292141    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292143    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292146    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292148    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:44:10.295106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292151    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292154    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292156    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292159    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292162    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292165    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292167    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292170    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292173    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292175    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292178    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292180    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292183    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292186    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292188    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292192    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292198    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292655    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292662    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:44:10.295637 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292665    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292668    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292672    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292675    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292678    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292681    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292684    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292687    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292690    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292693    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292696    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292698    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292701    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292704    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292707    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292709    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292712    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292715    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292719    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292723    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:44:10.296168 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292727    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292730    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292734    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292736    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292739    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292742    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292745    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292747    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292750    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292754    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292756    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292759    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292762    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292765    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292768    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292772    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292774    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292777    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292780    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292783    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:44:10.296691 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292785    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292788    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292790    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292793    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292795    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292798    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292801    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292803    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292806    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292809    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292811    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292814    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292816    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292820    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292822    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292825    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292828    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292830    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292833    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292835    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:44:10.297203 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292839    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292842    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292845    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292848    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292850    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292853    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292856    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292858    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292861    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292864    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292867    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292870    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292872    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292875    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292877    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292880    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292882    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292885    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292888    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292890    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:44:10.297774 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292893    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292895    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292898    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292901    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292903    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292906    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292909    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292913    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292915    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292918    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292921    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292923    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292930    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292934    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292938    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292941    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.292945    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293019    2552 flags.go:64] FLAG: --address="0.0.0.0"
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293027    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293034    2552 flags.go:64] FLAG: --anonymous-auth="true"
Jul 14 00:44:10.298289 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293039    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293045    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293048    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293053    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293057    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293061    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293064    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293068    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293071    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293075    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293078    2552 flags.go:64] FLAG: --cgroup-root=""
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293081    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293085    2552 flags.go:64] FLAG: --client-ca-file=""
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293087    2552 flags.go:64] FLAG: --cloud-provider="external"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293091    2552 flags.go:64] FLAG: --cluster-dns="[]"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293095    2552 flags.go:64] FLAG: --cluster-domain=""
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293098    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293102    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293105    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293109    2552 flags.go:64] FLAG: --container-log-max-files="5"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293113    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293116    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293119    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 14 00:44:10.298830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293123    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293128    2552 flags.go:64] FLAG: --contention-profiling="false"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293131    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293134    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293138    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293141    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293145    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293149    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293152    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293155    2552 flags.go:64] FLAG: --enable-load-reader="false"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293158    2552 flags.go:64] FLAG: --enable-server="true"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293162    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293167    2552 flags.go:64] FLAG: --event-burst="100"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293170    2552 flags.go:64] FLAG: --event-qps="50"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293173    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293176    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293179    2552 flags.go:64] FLAG: --eviction-hard=""
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293184    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293187    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293190    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293193    2552 flags.go:64] FLAG: --eviction-soft=""
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293196    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293199    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293202    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293205    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 14 00:44:10.299447 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293209    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293212    2552 flags.go:64] FLAG: --fail-swap-on="true"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293215    2552 flags.go:64] FLAG: --feature-gates=""
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293219    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293222    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293225    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293229    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293232    2552 flags.go:64] FLAG: --healthz-port="10248"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293237    2552 flags.go:64] FLAG: --help="false"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293241    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293245    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293248    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293251    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293254    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293258    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293262    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293265    2552 flags.go:64] FLAG: --image-service-endpoint=""
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293268    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293271    2552 flags.go:64] FLAG: --kube-api-burst="100"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293275    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293279    2552 flags.go:64] FLAG: --kube-api-qps="50"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293282    2552 flags.go:64] FLAG: --kube-reserved=""
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293285    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293288    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 14 00:44:10.300120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293291    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293294    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293298    2552 flags.go:64] FLAG: --lock-file=""
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293301    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293304    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293307    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293312    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293315    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293318    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293321    2552 flags.go:64] FLAG: --logging-format="text"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293324    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293327    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293331    2552 flags.go:64] FLAG: --manifest-url=""
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293333    2552 flags.go:64] FLAG: --manifest-url-header=""
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293338    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293341    2552 flags.go:64] FLAG: --max-open-files="1000000"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293347    2552 flags.go:64] FLAG: --max-pods="110"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293354    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293357    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293360    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293363    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293366    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293369    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293373    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293382    2552 flags.go:64] FLAG: --node-status-max-images="50"
Jul 14 00:44:10.300756 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293385    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293388    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293391    2552 flags.go:64] FLAG: --pod-cidr=""
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293395    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:b04c0b1f73af4d021dd8f0b51bfeb302c24482c093de5f2c43654ab56c506926"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293400    2552 flags.go:64] FLAG: --pod-manifest-path=""
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293403    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293407    2552 flags.go:64] FLAG: --pods-per-core="0"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293410    2552 flags.go:64] FLAG: --port="10250"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293413    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293416    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-07329a535cfbb1ce7"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293419    2552 flags.go:64] FLAG: --qos-reserved=""
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293423    2552 flags.go:64] FLAG: --read-only-port="10255"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293426    2552 flags.go:64] FLAG: --register-node="true"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293429    2552 flags.go:64] FLAG: --register-with-taints=""
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293433    2552 flags.go:64] FLAG: --registry-burst="10"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293436    2552 flags.go:64] FLAG: --registry-qps="5"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293439    2552 flags.go:64] FLAG: --reserved-cpus=""
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293442    2552 flags.go:64] FLAG: --reserved-memory=""
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293446    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293449    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293452    2552 flags.go:64] FLAG: --rotate-certificates="false"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293455    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293458    2552 flags.go:64] FLAG: --runonce="false"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293463    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293466    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 14 00:44:10.301397 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293471    2552 flags.go:64] FLAG: --seccomp-default="false"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293474    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293477    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293480    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293483    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293487    2552 flags.go:64] FLAG: --storage-driver-password="root"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293490    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293493    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293496    2552 flags.go:64] FLAG: --storage-driver-user="root"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293499    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293502    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293506    2552 flags.go:64] FLAG: --system-cgroups=""
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293510    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293516    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293519    2552 flags.go:64] FLAG: --tls-cert-file=""
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293523    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293527    2552 flags.go:64] FLAG: --tls-min-version=""
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293531    2552 flags.go:64] FLAG: --tls-private-key-file=""
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293534    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293537    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293540    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293543    2552 flags.go:64] FLAG: --v="2"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293548    2552 flags.go:64] FLAG: --version="false"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293552    2552 flags.go:64] FLAG: --vmodule=""
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293556    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 14 00:44:10.302062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.293560    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293716    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293722    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293726    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293730    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293734    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293738    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293743    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293747    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293750    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293753    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293756    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293759    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293762    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293765    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293768    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293771    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293774    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293777    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:44:10.302707 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293780    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293783    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293786    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293789    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293792    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293795    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293797    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293800    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293803    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293806    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293808    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293811    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293814    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293816    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293819    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293822    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293825    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293828    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293832    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293834    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:44:10.303238 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293839    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293841    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293844    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293847    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293850    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293853    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293855    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293858    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293861    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293863    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293866    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293869    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293872    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293875    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293877    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293880    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293883    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293885    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293889    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293891    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:44:10.303841 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293894    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293896    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293900    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293905    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293908    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293911    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293914    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293917    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293919    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293922    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293927    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293929    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293933    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293936    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293939    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293942    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293944    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293947    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293949    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293952    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:44:10.304359 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293955    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293957    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293960    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293963    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293966    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293969    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293972    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293975    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293978    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293980    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293983    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293986    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293988    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293991    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293993    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293996    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.293999    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.294001    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.294004    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:44:10.304903 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.294007    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.294009    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.294967    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.302488    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.302507    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.302522    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.302528    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302609    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302615    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302618    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302621    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302625    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302628    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302633    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:44:10.305378 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302637    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302640    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302643    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302646    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302649    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302653    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302657    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302660    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302663    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302667    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302669    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302673    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302675    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302678    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302689    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302692    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302695    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302698    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302701    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:44:10.305768 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302703    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302708    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302712    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302716    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302720    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302723    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302727    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302729    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302733    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302735    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302738    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302741    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302745    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302749    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302753    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302756    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302758    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302761    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302764    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:44:10.306257 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302768    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302771    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302775    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302778    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302781    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302784    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302787    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302790    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302792    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302795    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302798    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302800    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302803    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302806    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302809    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302812    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302815    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302818    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302821    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302824    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:44:10.306749 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302826    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302829    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302832    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302835    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302838    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302840    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302843    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302845    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302848    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302851    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302853    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302856    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302858    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302862    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302864    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302867    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302871    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302873    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302876    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302878    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:44:10.307271 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302881    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302883    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302886    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302889    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302891    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302894    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302896    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302900    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302904    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302907    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302910    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302912    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302915    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.302918    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.302923    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:44:10.307804 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303051    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303056    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303059    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303063    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303066    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303068    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303072    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303074    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303077    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303080    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303083    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303087    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303090    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303093    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303095    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303098    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303101    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303104    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303107    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303109    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 14 00:44:10.308206 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303112    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303114    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303118    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303122    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303125    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303128    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303131    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303134    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303137    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303139    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303142    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303145    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303147    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303150    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303152    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303155    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303157    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303160    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303162    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303165    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 14 00:44:10.308740 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303168    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303170    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303172    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303175    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303178    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303181    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303185    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303188    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303192    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303194    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303197    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303200    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303203    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303206    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303208    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303211    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303214    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303216    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303219    2552 feature_gate.go:328] unrecognized feature gate: Example
Jul 14 00:44:10.309246 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303222    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303224    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303227    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303230    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303232    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303235    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303238    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303241    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303243    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303246    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303249    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303251    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303254    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303257    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303259    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303262    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303264    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303267    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303270    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303272    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 14 00:44:10.309751 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303275    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303277    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303280    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303283    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303285    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303288    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303291    2552 feature_gate.go:328] unrecognized feature gate: Example2
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303293    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303297    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303301    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303304    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303306    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303309    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303312    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303315    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303317    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303320    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303322    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303325    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 14 00:44:10.310501 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:10.303328    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 14 00:44:10.311429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.303333    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 14 00:44:10.311429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.303464    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 14 00:44:10.311429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.305796    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 14 00:44:10.311429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.307011    2552 server.go:1019] "Starting client certificate rotation"
Jul 14 00:44:10.311429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.307133    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 00:44:10.311429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.307177    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 14 00:44:10.311429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.309658    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 00:44:10.312961 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.312937    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 14 00:44:10.329322 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.329291    2552 log.go:25] "Validated CRI v1 runtime API"
Jul 14 00:44:10.331649 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.331623    2552 log.go:25] "Validated CRI v1 image API"
Jul 14 00:44:10.332465 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.332436    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 14 00:44:10.336649 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.336615    2552 fs.go:135] Filesystem UUIDs: map[2e76e3b7-1edf-46c7-a570-ccad011372ce:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ffb4b750-8073-4bb6-8f21-943a66eaf2c6:/dev/nvme0n1p3]
Jul 14 00:44:10.336649 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.336645    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 14 00:44:10.339252 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.339227    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 00:44:10.341542 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.341429    2552 manager.go:216] Machine: {Timestamp:2026-07-14 00:44:10.340910868 +0000 UTC m=+0.321995524 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3201082 MemoryCapacity:33164500992 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec20deb9e3c09089429e99d956401fa1 SystemUUID:ec20deb9-e3c0-9089-429e-99d956401fa1 BootID:ca9777de-e981-4866-9892-56a5cbade08c Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582250496 Type:vfs Inodes:4048401 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6107136 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:19:89:fb:69:71 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:19:89:fb:69:71 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:b2:18:15:66:42:19 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164500992 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 14 00:44:10.341542 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.341536    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 14 00:44:10.341675 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.341641    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.126.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260702-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 14 00:44:10.345049 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.345016    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 14 00:44:10.345202 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.345053    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-125.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 14 00:44:10.345866 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.345855    2552 topology_manager.go:138] "Creating topology manager with none policy"
Jul 14 00:44:10.345909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.345869    2552 container_manager_linux.go:309] "Creating device plugin manager"
Jul 14 00:44:10.345909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.345877    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 00:44:10.345909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.345889    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 14 00:44:10.347266 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.347254    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 14 00:44:10.350170 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350156    2552 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 00:44:10.350308 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350297    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 14 00:44:10.350359 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350333    2552 kubelet.go:486] "Attempting to sync node with API server"
Jul 14 00:44:10.350406 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350361    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 14 00:44:10.350406 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350380    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 14 00:44:10.350406 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350390    2552 kubelet.go:398] "Adding apiserver pod source"
Jul 14 00:44:10.350406 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350401    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 14 00:44:10.350885 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350872    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 00:44:10.350953 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.350894    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 14 00:44:10.353451 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.353423    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 14 00:44:10.356118 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.356069    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 14 00:44:10.356263 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.356143    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 14 00:44:10.358300 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358283    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358304    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358310    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358316    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358325    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358332    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358341    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358347    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358356    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 14 00:44:10.358365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358362    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 14 00:44:10.358662 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358378    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 14 00:44:10.358662 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358387    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 14 00:44:10.358662 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358413    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 14 00:44:10.358662 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.358419    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 14 00:44:10.361390 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.361363    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 14 00:44:10.361390 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.361368    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 14 00:44:10.361512 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.361392    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-125.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 14 00:44:10.361951 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.361938    2552 server.go:1268] "Started kubelet"
Jul 14 00:44:10.362059 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.362036    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvrx5"
Jul 14 00:44:10.362120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.362035    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 14 00:44:10.362183 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.362043    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 14 00:44:10.362231 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.362216    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 14 00:44:10.362874 ip-10-0-129-125 systemd[1]: Started Kubernetes Kubelet.
Jul 14 00:44:10.364248 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.364236    2552 server.go:310] "Adding debug handlers to kubelet server"
Jul 14 00:44:10.366164 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.366145    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 14 00:44:10.368128 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.368101    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 14 00:44:10.368399 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.368381    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-tvrx5"
Jul 14 00:44:10.369322 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.368391    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-125.ec2.internal.18c2015121b93cc1  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-125.ec2.internal,UID:ip-10-0-129-125.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-125.ec2.internal,},FirstTimestamp:2026-07-14 00:44:10.361904321 +0000 UTC m=+0.342988975,LastTimestamp:2026-07-14 00:44:10.361904321 +0000 UTC m=+0.342988975,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-125.ec2.internal,}"
Jul 14 00:44:10.371270 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.371245    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 14 00:44:10.371270 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.371259    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 14 00:44:10.371426 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.371329    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:10.371862 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.371842    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 14 00:44:10.371945 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.371875    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 14 00:44:10.372034 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.371841    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 14 00:44:10.372145 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372131    2552 reconstruct.go:97] "Volume reconstruction finished"
Jul 14 00:44:10.372145 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372144    2552 reconciler.go:29] "Reconciler: start to sync state"
Jul 14 00:44:10.372699 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372679    2552 factory.go:153] Registering CRI-O factory
Jul 14 00:44:10.372800 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372705    2552 factory.go:223] Registration of the crio container factory successfully
Jul 14 00:44:10.372800 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372764    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 14 00:44:10.372800 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372775    2552 factory.go:55] Registering systemd factory
Jul 14 00:44:10.372800 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372786    2552 factory.go:223] Registration of the systemd container factory successfully
Jul 14 00:44:10.372962 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372813    2552 factory.go:103] Registering Raw factory
Jul 14 00:44:10.372962 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.372830    2552 manager.go:1196] Started watching for new ooms in manager
Jul 14 00:44:10.373404 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.373385    2552 manager.go:349] Starting recovery of all containers
Jul 14 00:44:10.375525 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.375498    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-125.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 14 00:44:10.375668 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.375643    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 14 00:44:10.389066 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.389042    2552 manager.go:354] Recovery completed
Jul 14 00:44:10.393792 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.393646    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:44:10.396170 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.396136    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:44:10.396278 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.396178    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:44:10.396278 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.396191    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:44:10.396701 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.396688    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 14 00:44:10.396701 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.396700    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 14 00:44:10.396842 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.396707    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:44:10.396842 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.396721    2552 state_mem.go:36] "Initialized new in-memory state store"
Jul 14 00:44:10.397487 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.397470    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:44:10.397555 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.397504    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:44:10.397555 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.397519    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:44:10.397922 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.397908    2552 policy_none.go:49] "None policy: Start"
Jul 14 00:44:10.397973 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.397926    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 14 00:44:10.397973 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.397939    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 14 00:44:10.399068 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.399055    2552 policy_none.go:47] "Start"
Jul 14 00:44:10.434287 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.434261    2552 manager.go:337] "Starting Device Plugin manager"
Jul 14 00:44:10.434423 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.434328    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 14 00:44:10.434423 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.434342    2552 server.go:87] "Starting device plugin registration server"
Jul 14 00:44:10.434734 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.434716    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 14 00:44:10.434855 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.434734    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 14 00:44:10.434855 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.434817    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 14 00:44:10.434965 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.434928    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 14 00:44:10.434965 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.434937    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 14 00:44:10.435716 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.435689    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 14 00:44:10.435807 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.435761    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:10.516659 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.516619    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 14 00:44:10.517963 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.517942    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 14 00:44:10.518061 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.517996    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 14 00:44:10.518061 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.518027    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 14 00:44:10.518157 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.518091    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 14 00:44:10.520292 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.520267    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:44:10.535264 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.535181    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:44:10.536198 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.536175    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:44:10.536325 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.536207    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:44:10.536325 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.536217    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:44:10.536325 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.536253    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.542166 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.542141    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.542237 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.542172    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-125.ec2.internal\": node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:10.558874 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.558846    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:10.619429 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.619381    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"]
Jul 14 00:44:10.619620 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.619472    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:44:10.620686 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.620664    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:44:10.620812 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.620704    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:44:10.620812 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.620716    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:44:10.622131 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.622119    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:44:10.622256 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.622234    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.622316 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.622277    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:44:10.622916 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.622903    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:44:10.622985 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.622934    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:44:10.622985 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.622946    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:44:10.623049 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.622907    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:44:10.623049 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.623014    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:44:10.623049 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.623045    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:44:10.624359 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.624343    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.624414 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.624372    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 14 00:44:10.625093 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.625077    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientMemory"
Jul 14 00:44:10.625197 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.625111    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasNoDiskPressure"
Jul 14 00:44:10.625197 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.625122    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeHasSufficientPID"
Jul 14 00:44:10.650663 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.650636    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.655789 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.655770    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-125.ec2.internal\" not found" node="ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.659406 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.659391    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:10.675280 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.675243    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7bc0ce05c4438b5438162ca5388e48b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"d7bc0ce05c4438b5438162ca5388e48b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.675439 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.675296    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7bc0ce05c4438b5438162ca5388e48b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"d7bc0ce05c4438b5438162ca5388e48b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.675439 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.675324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7e5f961bb90f8432e41003971876c05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"e7e5f961bb90f8432e41003971876c05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.760300 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.760260    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:10.775726 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.775691    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7bc0ce05c4438b5438162ca5388e48b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"d7bc0ce05c4438b5438162ca5388e48b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.775876 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.775733    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7bc0ce05c4438b5438162ca5388e48b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"d7bc0ce05c4438b5438162ca5388e48b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.775876 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.775788    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d7bc0ce05c4438b5438162ca5388e48b-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"d7bc0ce05c4438b5438162ca5388e48b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.775876 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.775794    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/d7bc0ce05c4438b5438162ca5388e48b-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal\" (UID: \"d7bc0ce05c4438b5438162ca5388e48b\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.775876 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.775828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7e5f961bb90f8432e41003971876c05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"e7e5f961bb90f8432e41003971876c05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.775876 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.775852    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/e7e5f961bb90f8432e41003971876c05-config\") pod \"kube-apiserver-proxy-ip-10-0-129-125.ec2.internal\" (UID: \"e7e5f961bb90f8432e41003971876c05\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.860649 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.860556    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:10.953099 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.953057    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.959910 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:10.959890    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:10.961046 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:10.961029    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.064859 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.064808    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.171546 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.171449    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.227049 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.227009    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:44:11.274146 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.274108    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.307980 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.307948    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 14 00:44:11.308538 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.308127    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 00:44:11.308538 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.308144    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 14 00:44:11.373439 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.373403    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-13 00:39:10 +0000 UTC" deadline="2028-04-18 11:36:43.902165986 +0000 UTC"
Jul 14 00:44:11.373439 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.373436    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15466h52m32.528735783s"
Jul 14 00:44:11.374586 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.374564    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.474953 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.474926    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.574086 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:11.574041    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd7bc0ce05c4438b5438162ca5388e48b.slice/crio-5768967a8443dcaed3646ceb6e2af3ef886c1b7bb5f288e0693c9dea618dbdd9 WatchSource:0}: Error finding container 5768967a8443dcaed3646ceb6e2af3ef886c1b7bb5f288e0693c9dea618dbdd9: Status 404 returned error can't find the container with id 5768967a8443dcaed3646ceb6e2af3ef886c1b7bb5f288e0693c9dea618dbdd9
Jul 14 00:44:11.574456 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:11.574435    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7e5f961bb90f8432e41003971876c05.slice/crio-afca08981636a43f6deb4c5c4c4f7970582f352439e85d968d719b078a73c1fc WatchSource:0}: Error finding container afca08981636a43f6deb4c5c4c4f7970582f352439e85d968d719b078a73c1fc: Status 404 returned error can't find the container with id afca08981636a43f6deb4c5c4c4f7970582f352439e85d968d719b078a73c1fc
Jul 14 00:44:11.575593 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.575571    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.578937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.578923    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:44:11.586154 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.586130    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:44:11.676807 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.676765    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.777871 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.777805    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.877992 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:11.877953    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-125.ec2.internal\" not found"
Jul 14 00:44:11.887751 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.887716    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:44:11.972592 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.972556    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:11.989394 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.989359    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 00:44:11.990680 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:11.990659    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal"
Jul 14 00:44:12.001086 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.001060    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 14 00:44:12.285884 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.285857    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:44:12.353901 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.353866    2552 apiserver.go:52] "Watching apiserver"
Jul 14 00:44:12.363242 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.363178    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 14 00:44:12.363738 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.363707    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-node-tuning-operator/tuned-2r2g4","openshift-dns/node-resolver-gbfs8","openshift-image-registry/node-ca-wq6ck","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal","openshift-multus/multus-jjgh9","openshift-multus/network-metrics-daemon-7bcw7","openshift-network-diagnostics/network-check-target-jxflr","openshift-network-operator/iptables-alerter-fg6nz","kube-system/global-pull-secret-syncer-w5xkv","kube-system/konnectivity-agent-xx6fm","kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc","openshift-multus/multus-additional-cni-plugins-4szch","openshift-ovn-kubernetes/ovnkube-node-k45vj"]
Jul 14 00:44:12.365577 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.365550    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.367216 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.367191    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.368160 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.368138    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 14 00:44:12.368160 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.368157    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 14 00:44:12.368324 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.368146    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-lqtgc\""
Jul 14 00:44:12.368851 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.368575    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.369606 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.369581    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 14 00:44:12.369707 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.369569    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-6zfm6\""
Jul 14 00:44:12.369769 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.369723    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 14 00:44:12.370140 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.369957    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 14 00:44:12.370140 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.370003    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 14 00:44:12.370140 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.370105    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.370774 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.370750    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 14 00:44:12.370919 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.370891    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 14 00:44:12.371137 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.371115    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 14 00:44:12.371208 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.371143    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 14 00:44:12.371433 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.371418    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-j7ppb\""
Jul 14 00:44:12.371536 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.371487    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.372426 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.372404    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 14 00:44:12.373116 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.373095    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:12.373228 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.373188    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:12.373369 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.373293    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374210    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-tgw6v\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374223    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374327    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374447    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374450    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374577    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374624    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.374684    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374722    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 14 00:44:12.374902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.374794    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-76jkd\""
Jul 14 00:44:12.375925 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.375906    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:12.376027 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.375978    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:12.377187 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.377168    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.380138 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.380113    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-t9jtq\""
Jul 14 00:44:12.380383 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.380365    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 14 00:44:12.380797 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.380777    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 14 00:44:12.381635 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.381550    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.383459 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.383433    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.383740 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.383718    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 14 00:44:12.384084 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.384063    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 14 00:44:12.384217 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.384195    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-2kftk\""
Jul 14 00:44:12.384837 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.384820    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.385178 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385151    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-var-lib-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385337 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385312    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-etc-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385458 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385346    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-cni-netd\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385458 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385371    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-env-overrides\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385458 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385396    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-system-cni-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.385458 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385423    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-cni-bin\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385475    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bae5b906-8455-4113-ba0b-5846d7b67f9e-agent-certs\") pod \"konnectivity-agent-xx6fm\" (UID: \"bae5b906-8455-4113-ba0b-5846d7b67f9e\") " pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385501    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-ovn\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385526    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovnkube-script-lib\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385551    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a13d6513-ea94-485d-adf6-532636f8112c-serviceca\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385601    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-run-netns\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385629    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-node-log\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385652    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-log-socket\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385674    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-run-ovn-kubernetes\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385699    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-cnibin\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385719    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-os-release\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.385752 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385746    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-cni-multus\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385772    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a13d6513-ea94-485d-adf6-532636f8112c-host\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385796    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-systemd\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385820    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovn-node-metrics-cert\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385844    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-netns\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385848    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385869    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-kubelet\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385894    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-hostroot\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385919    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-etc-kubernetes\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385943    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lbjr6\" (UniqueName: \"kubernetes.io/projected/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-kube-api-access-lbjr6\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385975    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-socket-dir-parent\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.385999    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-multus-certs\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386027    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-system-cni-dir\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386053    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386080    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386108    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c47h2\" (UniqueName: \"kubernetes.io/projected/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-kube-api-access-c47h2\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386134    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-systemd-units\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.386419 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386156    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386180    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnkmz\" (UniqueName: \"kubernetes.io/projected/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-kube-api-access-cnkmz\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386271    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-cni-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386296    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-k8s-cni-cncf-io\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386341    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386376    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zbg2c\" (UniqueName: \"kubernetes.io/projected/a13d6513-ea94-485d-adf6-532636f8112c-kube-api-access-zbg2c\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386425    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-kubelet\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386469    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-cni-bin\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386506    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-cni-binary-copy\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386532    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-conf-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386627    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-daemon-config\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386663    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cnibin\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386687    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-slash\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386720    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovnkube-config\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386764    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386771    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rl4j7\" (UniqueName: \"kubernetes.io/projected/79b4894c-cac8-408b-ae28-e3c6788ba884-kube-api-access-rl4j7\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:12.387440 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386818    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bae5b906-8455-4113-ba0b-5846d7b67f9e-konnectivity-ca\") pod \"konnectivity-agent-xx6fm\" (UID: \"bae5b906-8455-4113-ba0b-5846d7b67f9e\") " pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386842    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386872    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386916    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-zk8l5\""
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386844    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-os-release\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.386974    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cni-binary-copy\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.387051    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.387275    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.387306    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.387474    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 14 00:44:12.388261 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.387510    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-cvglv\""
Jul 14 00:44:12.405528 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.405495    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-nnjwj"
Jul 14 00:44:12.413500 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.413477    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-nnjwj"
Jul 14 00:44:12.473570 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.473540    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 14 00:44:12.488012 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.487961    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysconfig\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.488224 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488021    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-multus-certs\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.488224 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488073    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.488224 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488109    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnkmz\" (UniqueName: \"kubernetes.io/projected/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-kube-api-access-cnkmz\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.488224 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-k8s-cni-cncf-io\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.488224 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488178    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:12.488224 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488209    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-sys-fs\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488246    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-kubernetes\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488279    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysctl-conf\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488317    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-cni-bin\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488355    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-cni-binary-copy\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488384    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d727cbd0-53be-43d2-8798-f91cf7d73d0a-dbus\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488416    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44023eff-43a4-4016-a62c-f10e986d2a18-hosts-file\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488468    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v85bx\" (UniqueName: \"kubernetes.io/projected/0c9a2d62-5304-4f70-a513-519d570f32ad-kube-api-access-v85bx\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488502    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-slash\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.488547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488536    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cni-binary-copy\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488567    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-system-cni-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488623    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-tuned\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488667    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c9a2d62-5304-4f70-a513-519d570f32ad-iptables-alerter-script\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488700    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c9a2d62-5304-4f70-a513-519d570f32ad-host-slash\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488776    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-systemd\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488804    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-run\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488840    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a13d6513-ea94-485d-adf6-532636f8112c-serviceca\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488875    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-node-log\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488907    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-run-ovn-kubernetes\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.488943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488940    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-cnibin\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488969    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-os-release\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.488991    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-multus-certs\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489002    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489036    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d95d5b13-fac3-43ae-833d-2d52f56c3f68-tmp\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489114    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a13d6513-ea94-485d-adf6-532636f8112c-host\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489187    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-netns\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489230    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-kubelet\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489234    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489265    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-etc-kubernetes\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489298    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489338    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-k8s-cni-cncf-io\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.489437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489370    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-slash\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.489491    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.489615    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:12.989561074 +0000 UTC m=+2.970645724 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489656    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-cni-binary-copy\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489816    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-cnibin\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489922    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-os-release\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489938    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-etc-kubernetes\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489965    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-run-ovn-kubernetes\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489985    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/a13d6513-ea94-485d-adf6-532636f8112c-serviceca\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489299    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-cni-bin\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489343    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c47h2\" (UniqueName: \"kubernetes.io/projected/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-kube-api-access-c47h2\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490016    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/a13d6513-ea94-485d-adf6-532636f8112c-host\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.490036 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.489930    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-node-log\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490052    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-run-netns\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490064    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-kubelet\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490093    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhjj\" (UniqueName: \"kubernetes.io/projected/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-kube-api-access-bvhjj\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490105    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cni-binary-copy\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490137    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-system-cni-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490187    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-socket-dir-parent\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490227    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490264    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d727cbd0-53be-43d2-8798-f91cf7d73d0a-kubelet-config\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490299    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44023eff-43a4-4016-a62c-f10e986d2a18-tmp-dir\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490327    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-socket-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490267    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-socket-dir-parent\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490442    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-modprobe-d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490484    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-var-lib-kubelet\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490536    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-host\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.490567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490577    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-systemd-units\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490610    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490659    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490696    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-cni-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490728    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-device-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490775    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490806    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490843    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwm7d\" (UniqueName: \"kubernetes.io/projected/d95d5b13-fac3-43ae-833d-2d52f56c3f68-kube-api-access-dwm7d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490816    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-systemd-units\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490886    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zbg2c\" (UniqueName: \"kubernetes.io/projected/a13d6513-ea94-485d-adf6-532636f8112c-kube-api-access-zbg2c\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490893    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-cni-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490919    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-kubelet\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490952    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-conf-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.490978    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-daemon-config\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491026    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cnibin\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491096    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491127    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-sys\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.491392 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491158    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovnkube-config\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491191    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rl4j7\" (UniqueName: \"kubernetes.io/projected/79b4894c-cac8-408b-ae28-e3c6788ba884-kube-api-access-rl4j7\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491224    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bae5b906-8455-4113-ba0b-5846d7b67f9e-konnectivity-ca\") pod \"konnectivity-agent-xx6fm\" (UID: \"bae5b906-8455-4113-ba0b-5846d7b67f9e\") " pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491281    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-cnibin\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491320    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-os-release\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491370    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491402    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l7jsc\" (UniqueName: \"kubernetes.io/projected/44023eff-43a4-4016-a62c-f10e986d2a18-kube-api-access-l7jsc\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491628    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-tuning-conf-dir\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491794    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-kubelet\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491865    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/bae5b906-8455-4113-ba0b-5846d7b67f9e-konnectivity-ca\") pod \"konnectivity-agent-xx6fm\" (UID: \"bae5b906-8455-4113-ba0b-5846d7b67f9e\") " pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.491870    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-registration-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492085    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysctl-d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492180    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-os-release\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492182    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-daemon-config\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.492320 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492246    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-multus-conf-dir\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492367    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-var-lib-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492422    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-etc-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492458    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-cni-netd\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492504    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-cni-netd\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492518    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-var-lib-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492558    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-env-overrides\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492575    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-etc-openvswitch\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.492620    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-cni-bin\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493014    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.493055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493042    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-env-overrides\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493076    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-cni-bin\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493156    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-lib-modules\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493205    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-ovn\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493227    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovnkube-config\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493285    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-ovn\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovnkube-script-lib\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493331    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bae5b906-8455-4113-ba0b-5846d7b67f9e-agent-certs\") pod \"konnectivity-agent-xx6fm\" (UID: \"bae5b906-8455-4113-ba0b-5846d7b67f9e\") " pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.493548 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493407    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-run-netns\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493580    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-log-socket\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493656    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-cni-multus\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493691    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lbjr6\" (UniqueName: \"kubernetes.io/projected/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-kube-api-access-lbjr6\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493750    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-system-cni-dir\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493757    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493780    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-systemd\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovn-node-metrics-cert\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.493979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.493862    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-hostroot\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.494376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.494011    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovnkube-script-lib\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.494376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.494025    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-hostroot\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.494376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.494077    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-system-cni-dir\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.494376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.494146    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-run-systemd\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.494376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.494214    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-host-run-netns\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.494376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.494262    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-log-socket\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.494376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.494304    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-host-var-lib-cni-multus\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.498768 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.498738    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/bae5b906-8455-4113-ba0b-5846d7b67f9e-agent-certs\") pod \"konnectivity-agent-xx6fm\" (UID: \"bae5b906-8455-4113-ba0b-5846d7b67f9e\") " pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.499323 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.499293    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-ovn-node-metrics-cert\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.499427 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.499350    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c47h2\" (UniqueName: \"kubernetes.io/projected/58c65e1b-6b8e-499e-8009-1b5ae97ecf47-kube-api-access-c47h2\") pod \"multus-additional-cni-plugins-4szch\" (UID: \"58c65e1b-6b8e-499e-8009-1b5ae97ecf47\") " pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.499815 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.499793    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnkmz\" (UniqueName: \"kubernetes.io/projected/14cac740-f5b9-4a25-bcb2-3d12f82f8fce-kube-api-access-cnkmz\") pod \"ovnkube-node-k45vj\" (UID: \"14cac740-f5b9-4a25-bcb2-3d12f82f8fce\") " pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.500780 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.500757    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zbg2c\" (UniqueName: \"kubernetes.io/projected/a13d6513-ea94-485d-adf6-532636f8112c-kube-api-access-zbg2c\") pod \"node-ca-wq6ck\" (UID: \"a13d6513-ea94-485d-adf6-532636f8112c\") " pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.503157 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.502443    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rl4j7\" (UniqueName: \"kubernetes.io/projected/79b4894c-cac8-408b-ae28-e3c6788ba884-kube-api-access-rl4j7\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:12.503484 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.503434    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lbjr6\" (UniqueName: \"kubernetes.io/projected/eebeab35-9893-4fb3-b2e0-c5f66e9e520e-kube-api-access-lbjr6\") pod \"multus-jjgh9\" (UID: \"eebeab35-9893-4fb3-b2e0-c5f66e9e520e\") " pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.523551 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.523475    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" event={"ID":"e7e5f961bb90f8432e41003971876c05","Type":"ContainerStarted","Data":"afca08981636a43f6deb4c5c4c4f7970582f352439e85d968d719b078a73c1fc"}
Jul 14 00:44:12.524742 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.524707    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"d7bc0ce05c4438b5438162ca5388e48b","Type":"ContainerStarted","Data":"5768967a8443dcaed3646ceb6e2af3ef886c1b7bb5f288e0693c9dea618dbdd9"}
Jul 14 00:44:12.595241 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595150    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-device-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595241 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595200    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwm7d\" (UniqueName: \"kubernetes.io/projected/d95d5b13-fac3-43ae-833d-2d52f56c3f68-kube-api-access-dwm7d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595477 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595228    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595477 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595260    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-device-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595477 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-sys\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595477 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595335    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-sys\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595477 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595390    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l7jsc\" (UniqueName: \"kubernetes.io/projected/44023eff-43a4-4016-a62c-f10e986d2a18-kube-api-access-l7jsc\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.595477 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595448    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-registration-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595511    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysctl-d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595500    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-etc-selinux\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595550    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595566    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-registration-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595606    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-lib-modules\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595641    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysctl-d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595644    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysconfig\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.595676    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595696    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysconfig\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595698    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-sys-fs\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.595741    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret podName:d727cbd0-53be-43d2-8798-f91cf7d73d0a nodeName:}" failed. No retries permitted until 2026-07-14 00:44:13.095719422 +0000 UTC m=+3.076804073 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret") pod "global-pull-secret-syncer-w5xkv" (UID: "d727cbd0-53be-43d2-8798-f91cf7d73d0a") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595742    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-lib-modules\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595757    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-sys-fs\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595777    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-kubernetes\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.595804 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595808    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysctl-conf\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595840    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d727cbd0-53be-43d2-8798-f91cf7d73d0a-dbus\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595866    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44023eff-43a4-4016-a62c-f10e986d2a18-hosts-file\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595892    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v85bx\" (UniqueName: \"kubernetes.io/projected/0c9a2d62-5304-4f70-a513-519d570f32ad-kube-api-access-v85bx\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595926    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-tuned\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595948    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/d727cbd0-53be-43d2-8798-f91cf7d73d0a-dbus\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595894    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-kubernetes\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595968    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c9a2d62-5304-4f70-a513-519d570f32ad-iptables-alerter-script\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595993    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-sysctl-conf\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.595950    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/44023eff-43a4-4016-a62c-f10e986d2a18-hosts-file\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596004    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c9a2d62-5304-4f70-a513-519d570f32ad-host-slash\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596034    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/0c9a2d62-5304-4f70-a513-519d570f32ad-host-slash\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596053    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-systemd\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596083    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-run\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596119    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596128    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-systemd\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596144    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d95d5b13-fac3-43ae-833d-2d52f56c3f68-tmp\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596178    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-run\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.596594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596192    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhjj\" (UniqueName: \"kubernetes.io/projected/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-kube-api-access-bvhjj\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596196    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-kubelet-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596222    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596258    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d727cbd0-53be-43d2-8798-f91cf7d73d0a-kubelet-config\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44023eff-43a4-4016-a62c-f10e986d2a18-tmp-dir\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596312    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-socket-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-modprobe-d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596373    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-var-lib-kubelet\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596398    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-host\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596490    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-host\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596515    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-var-lib-kubelet\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596554    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/d727cbd0-53be-43d2-8798-f91cf7d73d0a-kubelet-config\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596529    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-modprobe-d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596659    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-socket-dir\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596720    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/0c9a2d62-5304-4f70-a513-519d570f32ad-iptables-alerter-script\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.597491 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.596901    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/44023eff-43a4-4016-a62c-f10e986d2a18-tmp-dir\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.599202 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.599166    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/d95d5b13-fac3-43ae-833d-2d52f56c3f68-tmp\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.599327 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.599280    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/d95d5b13-fac3-43ae-833d-2d52f56c3f68-etc-tuned\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.602244 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.602218    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:44:12.602244 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.602243    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:44:12.602411 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.602254    2552 projected.go:196] Error preparing data for projected volume kube-api-access-tvtw9 for pod openshift-network-diagnostics/network-check-target-jxflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:12.602411 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.602338    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9 podName:c658907e-9fbe-4320-a480-0ef6ffea21ea nodeName:}" failed. No retries permitted until 2026-07-14 00:44:13.102316896 +0000 UTC m=+3.083401537 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tvtw9" (UniqueName: "kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9") pod "network-check-target-jxflr" (UID: "c658907e-9fbe-4320-a480-0ef6ffea21ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:12.605122 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.605093    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l7jsc\" (UniqueName: \"kubernetes.io/projected/44023eff-43a4-4016-a62c-f10e986d2a18-kube-api-access-l7jsc\") pod \"node-resolver-gbfs8\" (UID: \"44023eff-43a4-4016-a62c-f10e986d2a18\") " pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.605227 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.605192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwm7d\" (UniqueName: \"kubernetes.io/projected/d95d5b13-fac3-43ae-833d-2d52f56c3f68-kube-api-access-dwm7d\") pod \"tuned-2r2g4\" (UID: \"d95d5b13-fac3-43ae-833d-2d52f56c3f68\") " pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.605949 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.605927    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v85bx\" (UniqueName: \"kubernetes.io/projected/0c9a2d62-5304-4f70-a513-519d570f32ad-kube-api-access-v85bx\") pod \"iptables-alerter-fg6nz\" (UID: \"0c9a2d62-5304-4f70-a513-519d570f32ad\") " pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.606495 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.606474    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhjj\" (UniqueName: \"kubernetes.io/projected/769eaa9d-7fbc-4ec7-9757-98f587fa52c3-kube-api-access-bvhjj\") pod \"aws-ebs-csi-driver-node-lbphc\" (UID: \"769eaa9d-7fbc-4ec7-9757-98f587fa52c3\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.679166 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.679129    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:12.689142 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.689111    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-4szch"
Jul 14 00:44:12.697716 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.697684    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-wq6ck"
Jul 14 00:44:12.703983 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.703952    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:12.711427 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.711400    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-jjgh9"
Jul 14 00:44:12.719683 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.719649    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gbfs8"
Jul 14 00:44:12.726850 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.726824    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-2r2g4"
Jul 14 00:44:12.734780 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.734753    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-fg6nz"
Jul 14 00:44:12.742782 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.742753    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc"
Jul 14 00:44:12.999022 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:12.998931    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:12.999198 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.999126    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:12.999258 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:12.999207    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:13.999183488 +0000 UTC m=+3.980268132 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:13.099920 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:13.099878    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:13.100110 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:13.100075    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:13.100196 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:13.100183    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret podName:d727cbd0-53be-43d2-8798-f91cf7d73d0a nodeName:}" failed. No retries permitted until 2026-07-14 00:44:14.1001561 +0000 UTC m=+4.081240748 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret") pod "global-pull-secret-syncer-w5xkv" (UID: "d727cbd0-53be-43d2-8798-f91cf7d73d0a") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:13.201051 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:13.201016    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:13.201228 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:13.201166    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:44:13.201228 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:13.201197    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:44:13.201228 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:13.201208    2552 projected.go:196] Error preparing data for projected volume kube-api-access-tvtw9 for pod openshift-network-diagnostics/network-check-target-jxflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:13.201333 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:13.201263    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9 podName:c658907e-9fbe-4320-a480-0ef6ffea21ea nodeName:}" failed. No retries permitted until 2026-07-14 00:44:14.201247729 +0000 UTC m=+4.182332370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvtw9" (UniqueName: "kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9") pod "network-check-target-jxflr" (UID: "c658907e-9fbe-4320-a480-0ef6ffea21ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:13.411172 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:13.411092    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 14 00:44:13.416527 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:13.416478    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 00:39:12 +0000 UTC" deadline="2028-01-21 13:47:10.673145347 +0000 UTC"
Jul 14 00:44:13.416527 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:13.416526    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13357h2m57.256623866s"
Jul 14 00:44:13.518573 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:13.518529    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:13.518748 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:13.518658    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:13.552012 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.551978    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13d6513_ea94_485d_adf6_532636f8112c.slice/crio-49f1109640d0beeccfd226754d9b27b2b9fbff5f5bf3caddd0dcff5c63e6c537 WatchSource:0}: Error finding container 49f1109640d0beeccfd226754d9b27b2b9fbff5f5bf3caddd0dcff5c63e6c537: Status 404 returned error can't find the container with id 49f1109640d0beeccfd226754d9b27b2b9fbff5f5bf3caddd0dcff5c63e6c537
Jul 14 00:44:13.556482 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.556410    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae5b906_8455_4113_ba0b_5846d7b67f9e.slice/crio-d83156d3bdd33d399cd4a52c19c2c304012e621ad58e50fb5bac11fd3f7caed6 WatchSource:0}: Error finding container d83156d3bdd33d399cd4a52c19c2c304012e621ad58e50fb5bac11fd3f7caed6: Status 404 returned error can't find the container with id d83156d3bdd33d399cd4a52c19c2c304012e621ad58e50fb5bac11fd3f7caed6
Jul 14 00:44:13.559009 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.558978    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeebeab35_9893_4fb3_b2e0_c5f66e9e520e.slice/crio-0f25d0c0b8321022a14185ca1cf6e11b27fc8e6ffe5a93041e99c4fc0c59f009 WatchSource:0}: Error finding container 0f25d0c0b8321022a14185ca1cf6e11b27fc8e6ffe5a93041e99c4fc0c59f009: Status 404 returned error can't find the container with id 0f25d0c0b8321022a14185ca1cf6e11b27fc8e6ffe5a93041e99c4fc0c59f009
Jul 14 00:44:13.560686 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.560566    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95d5b13_fac3_43ae_833d_2d52f56c3f68.slice/crio-e7b595162940fb854e38d832f4e043ea3c4da0757141b8b5f6511331fa1c6932 WatchSource:0}: Error finding container e7b595162940fb854e38d832f4e043ea3c4da0757141b8b5f6511331fa1c6932: Status 404 returned error can't find the container with id e7b595162940fb854e38d832f4e043ea3c4da0757141b8b5f6511331fa1c6932
Jul 14 00:44:13.561416 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.561389    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c9a2d62_5304_4f70_a513_519d570f32ad.slice/crio-22233ca61b73ba605844bb3621c18618b31f8d1780b258f70622956ee1f4ea9e WatchSource:0}: Error finding container 22233ca61b73ba605844bb3621c18618b31f8d1780b258f70622956ee1f4ea9e: Status 404 returned error can't find the container with id 22233ca61b73ba605844bb3621c18618b31f8d1780b258f70622956ee1f4ea9e
Jul 14 00:44:13.564421 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.564395    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58c65e1b_6b8e_499e_8009_1b5ae97ecf47.slice/crio-06d9b96478b5f25c46fb41c071d121068d311ffedf5f4eb52f6d0632f55ab6d3 WatchSource:0}: Error finding container 06d9b96478b5f25c46fb41c071d121068d311ffedf5f4eb52f6d0632f55ab6d3: Status 404 returned error can't find the container with id 06d9b96478b5f25c46fb41c071d121068d311ffedf5f4eb52f6d0632f55ab6d3
Jul 14 00:44:13.565410 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.565375    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44023eff_43a4_4016_a62c_f10e986d2a18.slice/crio-49cdf3b7023c758c4a248a150e3f40c104566aa9500ac5fcafcb92b913e0f9ff WatchSource:0}: Error finding container 49cdf3b7023c758c4a248a150e3f40c104566aa9500ac5fcafcb92b913e0f9ff: Status 404 returned error can't find the container with id 49cdf3b7023c758c4a248a150e3f40c104566aa9500ac5fcafcb92b913e0f9ff
Jul 14 00:44:13.566441 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.566320    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod769eaa9d_7fbc_4ec7_9757_98f587fa52c3.slice/crio-303aa35ee65d03833a419dae33c4ae746b9ffabf0aa83991295044485124531a WatchSource:0}: Error finding container 303aa35ee65d03833a419dae33c4ae746b9ffabf0aa83991295044485124531a: Status 404 returned error can't find the container with id 303aa35ee65d03833a419dae33c4ae746b9ffabf0aa83991295044485124531a
Jul 14 00:44:13.568106 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:13.568084    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14cac740_f5b9_4a25_bcb2_3d12f82f8fce.slice/crio-15e95b9b1a878102822d58be4a1a4ef0893757d012e52da62b8eb2bfac46bdf4 WatchSource:0}: Error finding container 15e95b9b1a878102822d58be4a1a4ef0893757d012e52da62b8eb2bfac46bdf4: Status 404 returned error can't find the container with id 15e95b9b1a878102822d58be4a1a4ef0893757d012e52da62b8eb2bfac46bdf4
Jul 14 00:44:14.009008 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.008680    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:14.009008 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.008981    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:14.009305 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.009072    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:16.009048558 +0000 UTC m=+5.990133205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:14.110288 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.110243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:14.110486 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.110425    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:14.110662 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.110518    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret podName:d727cbd0-53be-43d2-8798-f91cf7d73d0a nodeName:}" failed. No retries permitted until 2026-07-14 00:44:16.110496057 +0000 UTC m=+6.091580699 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret") pod "global-pull-secret-syncer-w5xkv" (UID: "d727cbd0-53be-43d2-8798-f91cf7d73d0a") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:14.212423 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.212374    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:14.212674 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.212565    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:44:14.212674 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.212606    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:44:14.212674 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.212621    2552 projected.go:196] Error preparing data for projected volume kube-api-access-tvtw9 for pod openshift-network-diagnostics/network-check-target-jxflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:14.212855 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.212687    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9 podName:c658907e-9fbe-4320-a480-0ef6ffea21ea nodeName:}" failed. No retries permitted until 2026-07-14 00:44:16.21266615 +0000 UTC m=+6.193750792 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvtw9" (UniqueName: "kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9") pod "network-check-target-jxflr" (UID: "c658907e-9fbe-4320-a480-0ef6ffea21ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:14.417154 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.416997    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-13 00:39:12 +0000 UTC" deadline="2028-01-20 05:02:36.365623368 +0000 UTC"
Jul 14 00:44:14.417154 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.417073    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13324h18m21.94855429s"
Jul 14 00:44:14.521385 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.521349    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:14.521605 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.521512    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:14.522008 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.521986    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:14.522126 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:14.522103    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:14.549520 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.548564    2552 generic.go:356] "Generic (PLEG): container finished" podID="d7bc0ce05c4438b5438162ca5388e48b" containerID="22f9bf13edb139f776185683affe971fd52485d71e04126f418701911a23ad01" exitCode=0
Jul 14 00:44:14.549735 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.549618    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"d7bc0ce05c4438b5438162ca5388e48b","Type":"ContainerDied","Data":"22f9bf13edb139f776185683affe971fd52485d71e04126f418701911a23ad01"}
Jul 14 00:44:14.556825 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.556793    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"15e95b9b1a878102822d58be4a1a4ef0893757d012e52da62b8eb2bfac46bdf4"}
Jul 14 00:44:14.564567 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.564532    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fg6nz" event={"ID":"0c9a2d62-5304-4f70-a513-519d570f32ad","Type":"ContainerStarted","Data":"22233ca61b73ba605844bb3621c18618b31f8d1780b258f70622956ee1f4ea9e"}
Jul 14 00:44:14.568534 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.568471    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjgh9" event={"ID":"eebeab35-9893-4fb3-b2e0-c5f66e9e520e","Type":"ContainerStarted","Data":"0f25d0c0b8321022a14185ca1cf6e11b27fc8e6ffe5a93041e99c4fc0c59f009"}
Jul 14 00:44:14.588439 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.588377    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xx6fm" event={"ID":"bae5b906-8455-4113-ba0b-5846d7b67f9e","Type":"ContainerStarted","Data":"d83156d3bdd33d399cd4a52c19c2c304012e621ad58e50fb5bac11fd3f7caed6"}
Jul 14 00:44:14.595191 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.595146    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" event={"ID":"e7e5f961bb90f8432e41003971876c05","Type":"ContainerStarted","Data":"d6ac80eeb9fd0b09122297e95fb9c6a62358af8760e4e678c5fe41c6f7413a7c"}
Jul 14 00:44:14.608767 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.608697    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2r2g4" event={"ID":"d95d5b13-fac3-43ae-833d-2d52f56c3f68","Type":"ContainerStarted","Data":"e7b595162940fb854e38d832f4e043ea3c4da0757141b8b5f6511331fa1c6932"}
Jul 14 00:44:14.625374 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.625267    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc" event={"ID":"769eaa9d-7fbc-4ec7-9757-98f587fa52c3","Type":"ContainerStarted","Data":"303aa35ee65d03833a419dae33c4ae746b9ffabf0aa83991295044485124531a"}
Jul 14 00:44:14.644259 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.644212    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gbfs8" event={"ID":"44023eff-43a4-4016-a62c-f10e986d2a18","Type":"ContainerStarted","Data":"49cdf3b7023c758c4a248a150e3f40c104566aa9500ac5fcafcb92b913e0f9ff"}
Jul 14 00:44:14.647076 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.647038    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerStarted","Data":"06d9b96478b5f25c46fb41c071d121068d311ffedf5f4eb52f6d0632f55ab6d3"}
Jul 14 00:44:14.657555 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:14.657515    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wq6ck" event={"ID":"a13d6513-ea94-485d-adf6-532636f8112c","Type":"ContainerStarted","Data":"49f1109640d0beeccfd226754d9b27b2b9fbff5f5bf3caddd0dcff5c63e6c537"}
Jul 14 00:44:15.518358 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:15.518321    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:15.518983 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:15.518465    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:15.668594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:15.668544    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" event={"ID":"d7bc0ce05c4438b5438162ca5388e48b","Type":"ContainerStarted","Data":"bb2495e945762c66b0aca504922527a064e9c607b18368b4b9a836164a51b654"}
Jul 14 00:44:15.683747 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:15.683681    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-125.ec2.internal" podStartSLOduration=3.6836557450000003 podStartE2EDuration="3.683655745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:14.611031431 +0000 UTC m=+4.592116095" watchObservedRunningTime="2026-07-14 00:44:15.683655745 +0000 UTC m=+5.664740413"
Jul 14 00:44:15.684434 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:15.684390    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-125.ec2.internal" podStartSLOduration=3.684379344 podStartE2EDuration="3.684379344s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:15.683825283 +0000 UTC m=+5.664909948" watchObservedRunningTime="2026-07-14 00:44:15.684379344 +0000 UTC m=+5.665464010"
Jul 14 00:44:16.029631 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:16.029589    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:16.029837 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.029802    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:16.029907 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.029875    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:20.029853652 +0000 UTC m=+10.010938296 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:16.134598 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:16.130601    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:16.143192 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.138880    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:16.143192 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.138993    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret podName:d727cbd0-53be-43d2-8798-f91cf7d73d0a nodeName:}" failed. No retries permitted until 2026-07-14 00:44:20.138965247 +0000 UTC m=+10.120049891 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret") pod "global-pull-secret-syncer-w5xkv" (UID: "d727cbd0-53be-43d2-8798-f91cf7d73d0a") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:16.232313 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:16.231478    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:16.232313 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.231702    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:44:16.232313 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.231725    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:44:16.232313 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.231739    2552 projected.go:196] Error preparing data for projected volume kube-api-access-tvtw9 for pod openshift-network-diagnostics/network-check-target-jxflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:16.232313 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.231817    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9 podName:c658907e-9fbe-4320-a480-0ef6ffea21ea nodeName:}" failed. No retries permitted until 2026-07-14 00:44:20.231799129 +0000 UTC m=+10.212883779 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvtw9" (UniqueName: "kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9") pod "network-check-target-jxflr" (UID: "c658907e-9fbe-4320-a480-0ef6ffea21ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:16.519776 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:16.519684    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:16.520251 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.519880    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:16.522282 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:16.522255    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:16.522422 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:16.522368    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:17.518788 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:17.518717    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:17.518969 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:17.518852    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:18.519139 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:18.519068    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:18.519598 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:18.519206    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:18.519598 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:18.519279    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:18.519598 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:18.519361    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:19.518824 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:19.518658    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:19.518824 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:19.518775    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:20.067880 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:20.067804    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:20.068374 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.067956    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:20.068374 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.068033    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:28.068010902 +0000 UTC m=+18.049095547 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:20.169253 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:20.169185    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:20.169442 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.169322    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:20.169442 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.169393    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret podName:d727cbd0-53be-43d2-8798-f91cf7d73d0a nodeName:}" failed. No retries permitted until 2026-07-14 00:44:28.169374069 +0000 UTC m=+18.150458722 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret") pod "global-pull-secret-syncer-w5xkv" (UID: "d727cbd0-53be-43d2-8798-f91cf7d73d0a") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:20.270360 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:20.270317    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:20.270555 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.270478    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:44:20.270555 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.270521    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:44:20.270555 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.270544    2552 projected.go:196] Error preparing data for projected volume kube-api-access-tvtw9 for pod openshift-network-diagnostics/network-check-target-jxflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:20.270715 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.270600    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9 podName:c658907e-9fbe-4320-a480-0ef6ffea21ea nodeName:}" failed. No retries permitted until 2026-07-14 00:44:28.270585394 +0000 UTC m=+18.251670033 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvtw9" (UniqueName: "kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9") pod "network-check-target-jxflr" (UID: "c658907e-9fbe-4320-a480-0ef6ffea21ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:20.525394 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:20.524875    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:20.525394 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.524999    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:20.525394 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:20.525215    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:20.525679 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:20.525414    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:21.519306 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:21.519260    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:21.519801 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:21.519391    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:22.523143 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:22.523110    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:22.523624 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:22.523238    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:22.523624 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:22.523245    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:22.523624 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:22.523392    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:23.518649 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:23.518615    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:23.518801 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:23.518724    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:24.521385 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:24.521360    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:24.521800 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:24.521457    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:24.521800 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:24.521562    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:24.521800 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:24.521674    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:25.518347 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:25.518311    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:25.518519 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:25.518422    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:26.518792 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:26.518757    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:26.519208 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:26.518799    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:26.519208 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:26.518895    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:26.519208 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:26.519031    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:27.518791 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:27.518751    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:27.519339 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:27.518896    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:28.129916 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:28.129872    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:28.130104 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.130028    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:28.130104 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.130098    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:44.130081051 +0000 UTC m=+34.111165690 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:28.230693 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:28.230661    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:28.230914 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.230815    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:28.230914 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.230888    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret podName:d727cbd0-53be-43d2-8798-f91cf7d73d0a nodeName:}" failed. No retries permitted until 2026-07-14 00:44:44.230873065 +0000 UTC m=+34.211957722 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret") pod "global-pull-secret-syncer-w5xkv" (UID: "d727cbd0-53be-43d2-8798-f91cf7d73d0a") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:28.331629 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:28.331593    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:28.331826 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.331772    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:44:28.331826 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.331794    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:44:28.331826 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.331805    2552 projected.go:196] Error preparing data for projected volume kube-api-access-tvtw9 for pod openshift-network-diagnostics/network-check-target-jxflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:28.331984 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.331876    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9 podName:c658907e-9fbe-4320-a480-0ef6ffea21ea nodeName:}" failed. No retries permitted until 2026-07-14 00:44:44.331849525 +0000 UTC m=+34.312934165 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvtw9" (UniqueName: "kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9") pod "network-check-target-jxflr" (UID: "c658907e-9fbe-4320-a480-0ef6ffea21ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:28.521524 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:28.521430    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:28.521954 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.521574    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:28.521954 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:28.521646    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:28.521954 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:28.521749    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:29.519315 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:29.519275    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:29.519527 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:29.519402    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:30.519515 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:30.519467    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:30.519927 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:30.519605    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:30.519927 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:30.519667    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:30.519927 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:30.519777    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:31.519218 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:31.519179    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:31.519397 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:31.519300    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:32.521695 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.520151    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:32.521695 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.521065    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:32.521695 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:32.521179    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:32.522484 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:32.520481    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:32.707926 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.707898    2552 generic.go:356] "Generic (PLEG): container finished" podID="58c65e1b-6b8e-499e-8009-1b5ae97ecf47" containerID="2fa05f63ea0694b7bb22cfd539cf0574086cef43ad46eedc7e6631816a0476fd" exitCode=0
Jul 14 00:44:32.708052 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.707977    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerDied","Data":"2fa05f63ea0694b7bb22cfd539cf0574086cef43ad46eedc7e6631816a0476fd"}
Jul 14 00:44:32.710277 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.710245    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-wq6ck" event={"ID":"a13d6513-ea94-485d-adf6-532636f8112c","Type":"ContainerStarted","Data":"e855a23118db9485e99b6d64900ee10557a62637ab4104a31239a04df51ad55b"}
Jul 14 00:44:32.712368 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.712354    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:44:32.712690 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.712670    2552 generic.go:356] "Generic (PLEG): container finished" podID="14cac740-f5b9-4a25-bcb2-3d12f82f8fce" containerID="e7a8d1d3288fb31c27c71b8aca146d41b7005d3de6addda4a456e716dc28ff47" exitCode=1
Jul 14 00:44:32.712768 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.712745    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"bc058b0bf8d2311b252f2bc3e4638167b24917a6ff899fb27bebc502d011cd6c"}
Jul 14 00:44:32.712837 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.712807    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerDied","Data":"e7a8d1d3288fb31c27c71b8aca146d41b7005d3de6addda4a456e716dc28ff47"}
Jul 14 00:44:32.712887 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.712838    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"a06ffbdb3990e2902df829c7a53ed179cc02f75c4d1ee88ce25ca1c041e35184"}
Jul 14 00:44:32.714084 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.714063    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-jjgh9" event={"ID":"eebeab35-9893-4fb3-b2e0-c5f66e9e520e","Type":"ContainerStarted","Data":"414abc871324c01b6a4a0a2d80ca2fe7167064964911a4da6b605b50b73cebdd"}
Jul 14 00:44:32.715662 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.715644    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-xx6fm" event={"ID":"bae5b906-8455-4113-ba0b-5846d7b67f9e","Type":"ContainerStarted","Data":"9a8c6c56ee9ae304e0bfdefdf4eb4a34c34fb6eca5143a210d40ff055c290bd2"}
Jul 14 00:44:32.716925 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.716909    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-2r2g4" event={"ID":"d95d5b13-fac3-43ae-833d-2d52f56c3f68","Type":"ContainerStarted","Data":"626069c65340521b353c7a3031e0dfa8962f41884fb12d19d87731219e53a6ce"}
Jul 14 00:44:32.718130 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.718102    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc" event={"ID":"769eaa9d-7fbc-4ec7-9757-98f587fa52c3","Type":"ContainerStarted","Data":"49d1e8616bbf02fdca11a392ee0406ea1617587e877df7f529e94ec93aa02c74"}
Jul 14 00:44:32.719329 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.719312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gbfs8" event={"ID":"44023eff-43a4-4016-a62c-f10e986d2a18","Type":"ContainerStarted","Data":"158c97cbf64d9ea80485aa78aab3921e3c35e368971e1ae1081546babb19b6f4"}
Jul 14 00:44:32.745167 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.745117    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-wq6ck" podStartSLOduration=2.960240251 podStartE2EDuration="21.745100362s" totalImagesPullingTime="18.784860111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:32.744685389 +0000 UTC m=+22.725770050" watchObservedRunningTime="2026-07-14 00:44:32.745100362 +0000 UTC m=+22.726185022"
Jul 14 00:44:32.760364 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.760319    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-jjgh9" podStartSLOduration=2.923881764 podStartE2EDuration="21.760302823s" totalImagesPullingTime="18.836421059s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:32.759886612 +0000 UTC m=+22.740971273" watchObservedRunningTime="2026-07-14 00:44:32.760302823 +0000 UTC m=+22.741387483"
Jul 14 00:44:32.774004 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.773949    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gbfs8" podStartSLOduration=3.01930196 podStartE2EDuration="21.773929765s" totalImagesPullingTime="18.754627805s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:32.773785351 +0000 UTC m=+22.754870012" watchObservedRunningTime="2026-07-14 00:44:32.773929765 +0000 UTC m=+22.755014427"
Jul 14 00:44:32.789841 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.789791    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-2r2g4" podStartSLOduration=3.009969095 podStartE2EDuration="21.789771821s" totalImagesPullingTime="18.779802726s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:32.789304808 +0000 UTC m=+22.770389469" watchObservedRunningTime="2026-07-14 00:44:32.789771821 +0000 UTC m=+22.770856481"
Jul 14 00:44:32.802853 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:32.802811    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-xx6fm" podStartSLOduration=10.879976267 podStartE2EDuration="21.802795836s" totalImagesPullingTime="10.922819569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:32.802602433 +0000 UTC m=+22.783687094" watchObservedRunningTime="2026-07-14 00:44:32.802795836 +0000 UTC m=+22.783880496"
Jul 14 00:44:33.519114 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:33.518899    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:33.519282 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:33.519198    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:33.724428 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:33.724403    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:44:33.725173 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:33.724871    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"2bc1474b657d071afa4c9c6766eeeed48eb058bb16e1ad07567ee8f1bb9da341"}
Jul 14 00:44:33.725173 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:33.724954    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"57e90ef36456a5ccff42d4c3feeec69bcb0391d1e31d83987ba3ea9e0bf39926"}
Jul 14 00:44:33.725173 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:33.725035    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"d778d2f83aa4c785cf47b4d5bcd37e77c83b69da7636c9bd9dde4bc56f22d76c"}
Jul 14 00:44:33.897609 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:33.897581    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 14 00:44:34.453656 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:34.453552    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-14T00:44:33.897599833Z","UUID":"80aaa0df-01db-45b4-af69-5617c0a84878","Handler":null,"Name":"","Endpoint":""}
Jul 14 00:44:34.455071 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:34.455043    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 14 00:44:34.455200 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:34.455084    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 14 00:44:34.519194 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:34.519163    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:34.519194 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:34.519180    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:34.519420 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:34.519296    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:34.519559 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:34.519452    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:34.729616 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:34.729521    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc" event={"ID":"769eaa9d-7fbc-4ec7-9757-98f587fa52c3","Type":"ContainerStarted","Data":"2804101a8bb740f70d0bc629a03487f04f95a2e95616eb7889bfde9736296923"}
Jul 14 00:44:34.731163 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:34.731133    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-fg6nz" event={"ID":"0c9a2d62-5304-4f70-a513-519d570f32ad","Type":"ContainerStarted","Data":"76e1921a4eb58372d46ff57da4863924e835c1a19eec70a6fb2177792f585e30"}
Jul 14 00:44:35.519250 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:35.519179    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:35.519462 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:35.519277    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:35.736219 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:35.735945    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:44:35.736825 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:35.736639    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"81eae110ad769bfc6a885ef240ca68ea51f467b7583627964c8c178dadecf926"}
Jul 14 00:44:35.738723 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:35.738703    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc" event={"ID":"769eaa9d-7fbc-4ec7-9757-98f587fa52c3","Type":"ContainerStarted","Data":"1dd3cc45d2bf9a107f3981666419c1f1a98ebbb9a2febfc01b5f99bf0a7ee3bf"}
Jul 14 00:44:35.755536 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:35.755474    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-fg6nz" podStartSLOduration=5.978286153 podStartE2EDuration="24.755456628s" totalImagesPullingTime="18.777170475s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:34.759558637 +0000 UTC m=+24.740643303" watchObservedRunningTime="2026-07-14 00:44:35.755456628 +0000 UTC m=+25.736541288"
Jul 14 00:44:35.755701 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:35.755639    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-lbphc" podStartSLOduration=3.574191913 podStartE2EDuration="24.755632193s" totalImagesPullingTime="21.18144028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:35.754394624 +0000 UTC m=+25.735479298" watchObservedRunningTime="2026-07-14 00:44:35.755632193 +0000 UTC m=+25.736716899"
Jul 14 00:44:36.518479 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:36.518436    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:36.518680 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:36.518455    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:36.518680 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:36.518605    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:36.518799 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:36.518707    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:37.518848 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:37.518770    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:37.519469 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:37.518885    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:37.561760 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:37.561727    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:37.562430 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:37.562407    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:37.744174 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:37.744146    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:37.744707 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:37.744684    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-xx6fm"
Jul 14 00:44:38.518886 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:38.518802    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:38.518886 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:38.518840    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:38.519396 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:38.518948    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:38.519458 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:38.519404    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:38.750376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:38.750348    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:44:38.750892 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:38.750800    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"e66312a0ea58d4203a9c9b56edaede1a408664b3fc2cbde6227c05732720d85b"}
Jul 14 00:44:39.518893 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:39.518708    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:39.519414 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:39.518968    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:39.754382 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:39.754349    2552 generic.go:356] "Generic (PLEG): container finished" podID="58c65e1b-6b8e-499e-8009-1b5ae97ecf47" containerID="b9f1bde1b9f83d80b62bc4eb306afea55f2ff5c33f2e0b55380ce5c8c92a046c" exitCode=0
Jul 14 00:44:39.754597 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:39.754426    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerDied","Data":"b9f1bde1b9f83d80b62bc4eb306afea55f2ff5c33f2e0b55380ce5c8c92a046c"}
Jul 14 00:44:39.754954 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:39.754927    2552 scope.go:117] "RemoveContainer" containerID="e7a8d1d3288fb31c27c71b8aca146d41b7005d3de6addda4a456e716dc28ff47"
Jul 14 00:44:40.519780 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.519749    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:40.520476 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:40.519879    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:40.520476 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.519990    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:40.520476 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:40.520117    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:40.691062 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.691033    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w5xkv"]
Jul 14 00:44:40.693851 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.693822    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxflr"]
Jul 14 00:44:40.694022 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.693940    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:40.694089 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:40.694053    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:40.694520 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.694485    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bcw7"]
Jul 14 00:44:40.760089 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.760020    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:44:40.760412 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.760389    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" event={"ID":"14cac740-f5b9-4a25-bcb2-3d12f82f8fce","Type":"ContainerStarted","Data":"f64d44d8bf8dd018e357d16296ff9e0b63f4e80474be78b564113cb1571282c9"}
Jul 14 00:44:40.760412 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.760405    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:40.760559 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.760423    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:40.760559 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:40.760510    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:40.760695 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.760677    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:40.760864 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.760848    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:40.761100 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.761081    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:40.761185 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:40.760826    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:40.776475 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.776299    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:40.776632 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.776577    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:44:40.786471 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:40.786426    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj" podStartSLOduration=10.962664738 podStartE2EDuration="29.78641123s" totalImagesPullingTime="18.823746492s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:40.784943492 +0000 UTC m=+30.766028153" watchObservedRunningTime="2026-07-14 00:44:40.78641123 +0000 UTC m=+30.767495890"
Jul 14 00:44:41.765948 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:41.765865    2552 generic.go:356] "Generic (PLEG): container finished" podID="58c65e1b-6b8e-499e-8009-1b5ae97ecf47" containerID="a33f8dc2841b6d189fb61738f2fabaad7b3168fa469de0fb010c2821821c4fa1" exitCode=0
Jul 14 00:44:41.766359 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:41.765943    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerDied","Data":"a33f8dc2841b6d189fb61738f2fabaad7b3168fa469de0fb010c2821821c4fa1"}
Jul 14 00:44:42.518709 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:42.518677    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:42.518709 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:42.518693    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:42.518975 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:42.518677    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:42.518975 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:42.518800    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:42.518975 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:42.518908    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:42.519116 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:42.518976    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:43.773095 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:43.773011    2552 generic.go:356] "Generic (PLEG): container finished" podID="58c65e1b-6b8e-499e-8009-1b5ae97ecf47" containerID="309d8ec693e277279bcfe1c5e1c4e1665213c9f1ac23a709578c6f906811f540" exitCode=0
Jul 14 00:44:43.773095 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:43.773048    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerDied","Data":"309d8ec693e277279bcfe1c5e1c4e1665213c9f1ac23a709578c6f906811f540"}
Jul 14 00:44:44.157041 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:44.157002    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:44.157230 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.157113    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:44.157230 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.157166    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:45:16.157150173 +0000 UTC m=+66.138234812 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 14 00:44:44.257368 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:44.257331    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:44.257576 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.257442    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:44.257576 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.257515    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret podName:d727cbd0-53be-43d2-8798-f91cf7d73d0a nodeName:}" failed. No retries permitted until 2026-07-14 00:45:16.257481638 +0000 UTC m=+66.238566277 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret") pod "global-pull-secret-syncer-w5xkv" (UID: "d727cbd0-53be-43d2-8798-f91cf7d73d0a") : object "kube-system"/"original-pull-secret" not registered
Jul 14 00:44:44.358671 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:44.358622    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:44.358835 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.358797    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 14 00:44:44.358835 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.358826    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 14 00:44:44.358960 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.358841    2552 projected.go:196] Error preparing data for projected volume kube-api-access-tvtw9 for pod openshift-network-diagnostics/network-check-target-jxflr: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:44.358960 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.358911    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9 podName:c658907e-9fbe-4320-a480-0ef6ffea21ea nodeName:}" failed. No retries permitted until 2026-07-14 00:45:16.358892404 +0000 UTC m=+66.339977046 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tvtw9" (UniqueName: "kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9") pod "network-check-target-jxflr" (UID: "c658907e-9fbe-4320-a480-0ef6ffea21ea") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 14 00:44:44.518931 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:44.518713    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:44.518931 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:44.518714    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:44.518931 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.518846    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-jxflr" podUID="c658907e-9fbe-4320-a480-0ef6ffea21ea"
Jul 14 00:44:44.519187 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:44.518724    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:44.519187 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.518941    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w5xkv" podUID="d727cbd0-53be-43d2-8798-f91cf7d73d0a"
Jul 14 00:44:44.519187 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:44.519055    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:44:45.362798 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.362768    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-125.ec2.internal" event="NodeReady"
Jul 14 00:44:45.363376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.362915    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 14 00:44:45.408939 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.408896    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-qqm2z"]
Jul 14 00:44:45.411264 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.411236    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-lkgk6"]
Jul 14 00:44:45.411487 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.411465    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.413045 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.413022    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.414058 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.414028    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 14 00:44:45.414058 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.414024    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 14 00:44:45.414230 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.414164    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rvlnc\""
Jul 14 00:44:45.415375 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.415350    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 14 00:44:45.415542 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.415387    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 14 00:44:45.415620 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.415562    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 14 00:44:45.415691 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.415660    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rd24t\""
Jul 14 00:44:45.421182 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.420895    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qqm2z"]
Jul 14 00:44:45.421605 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.421551    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lkgk6"]
Jul 14 00:44:45.568204 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.568163    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d478348-8510-4b2a-bb83-45abe05300d0-cert\") pod \"ingress-canary-lkgk6\" (UID: \"3d478348-8510-4b2a-bb83-45abe05300d0\") " pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.568384 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.568217    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fss4w\" (UniqueName: \"kubernetes.io/projected/3d478348-8510-4b2a-bb83-45abe05300d0-kube-api-access-fss4w\") pod \"ingress-canary-lkgk6\" (UID: \"3d478348-8510-4b2a-bb83-45abe05300d0\") " pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.568384 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.568273    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdd529a-053a-4f90-8e1f-88143f580ff9-config-volume\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.568384 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.568348    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdd529a-053a-4f90-8e1f-88143f580ff9-tmp-dir\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.568384 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.568379    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cmxg5\" (UniqueName: \"kubernetes.io/projected/7cdd529a-053a-4f90-8e1f-88143f580ff9-kube-api-access-cmxg5\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.568616 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.568406    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.669200 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.669104    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d478348-8510-4b2a-bb83-45abe05300d0-cert\") pod \"ingress-canary-lkgk6\" (UID: \"3d478348-8510-4b2a-bb83-45abe05300d0\") " pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.669200 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.669148    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fss4w\" (UniqueName: \"kubernetes.io/projected/3d478348-8510-4b2a-bb83-45abe05300d0-kube-api-access-fss4w\") pod \"ingress-canary-lkgk6\" (UID: \"3d478348-8510-4b2a-bb83-45abe05300d0\") " pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.669411 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.669203    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdd529a-053a-4f90-8e1f-88143f580ff9-config-volume\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.669411 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.669249    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdd529a-053a-4f90-8e1f-88143f580ff9-tmp-dir\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.669411 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.669272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cmxg5\" (UniqueName: \"kubernetes.io/projected/7cdd529a-053a-4f90-8e1f-88143f580ff9-kube-api-access-cmxg5\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.669547 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.669523    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.669734 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:45.669703    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:44:45.669845 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:45.669802    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:46.169779685 +0000 UTC m=+36.150864342 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:44:45.670390 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.670367    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7cdd529a-053a-4f90-8e1f-88143f580ff9-tmp-dir\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.670739 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.670717    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7cdd529a-053a-4f90-8e1f-88143f580ff9-config-volume\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.674230 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.674206    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/3d478348-8510-4b2a-bb83-45abe05300d0-cert\") pod \"ingress-canary-lkgk6\" (UID: \"3d478348-8510-4b2a-bb83-45abe05300d0\") " pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.678538 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.678371    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cmxg5\" (UniqueName: \"kubernetes.io/projected/7cdd529a-053a-4f90-8e1f-88143f580ff9-kube-api-access-cmxg5\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:45.678839 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.678802    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fss4w\" (UniqueName: \"kubernetes.io/projected/3d478348-8510-4b2a-bb83-45abe05300d0-kube-api-access-fss4w\") pod \"ingress-canary-lkgk6\" (UID: \"3d478348-8510-4b2a-bb83-45abe05300d0\") " pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.731480 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.731445    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-lkgk6"
Jul 14 00:44:45.890982 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:45.890940    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-lkgk6"]
Jul 14 00:44:45.896465 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:44:45.896435    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d478348_8510_4b2a_bb83_45abe05300d0.slice/crio-dfe859894a6944a041268b1bf3683929e4f7d7422e8a54f07b097627100cbfac WatchSource:0}: Error finding container dfe859894a6944a041268b1bf3683929e4f7d7422e8a54f07b097627100cbfac: Status 404 returned error can't find the container with id dfe859894a6944a041268b1bf3683929e4f7d7422e8a54f07b097627100cbfac
Jul 14 00:44:46.173993 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.173953    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:46.174177 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:46.174129    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:44:46.174234 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:46.174206    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:47.174190937 +0000 UTC m=+37.155275594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:44:46.518374 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.518283    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:44:46.518825 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.518285    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:44:46.518825 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.518291    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:44:46.520901 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.520878    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 00:44:46.520901 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.520893    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 00:44:46.521166 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.521145    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tbhhd\""
Jul 14 00:44:46.521253 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.521181    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 00:44:46.522253 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.522210    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cmcmt\""
Jul 14 00:44:46.522253 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.522239    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 00:44:46.782022 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:46.781941    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lkgk6" event={"ID":"3d478348-8510-4b2a-bb83-45abe05300d0","Type":"ContainerStarted","Data":"dfe859894a6944a041268b1bf3683929e4f7d7422e8a54f07b097627100cbfac"}
Jul 14 00:44:47.182201 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:47.182163    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:47.182376 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:47.182351    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:44:47.182465 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:47.182444    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:49.182421386 +0000 UTC m=+39.163506026 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:44:47.793140 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:47.793088    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerStarted","Data":"8c08c5a95b2a14ccbac1f07867178a05ac567a542579375eaff90af802f65cca"}
Jul 14 00:44:48.799363 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:48.799152    2552 generic.go:356] "Generic (PLEG): container finished" podID="58c65e1b-6b8e-499e-8009-1b5ae97ecf47" containerID="8c08c5a95b2a14ccbac1f07867178a05ac567a542579375eaff90af802f65cca" exitCode=0
Jul 14 00:44:48.799830 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:48.799248    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerDied","Data":"8c08c5a95b2a14ccbac1f07867178a05ac567a542579375eaff90af802f65cca"}
Jul 14 00:44:49.198161 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:49.198125    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:49.198357 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:49.198245    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:44:49.198357 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:49.198303    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:44:53.198283517 +0000 UTC m=+43.179368157 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:44:49.804360 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:49.804272    2552 generic.go:356] "Generic (PLEG): container finished" podID="58c65e1b-6b8e-499e-8009-1b5ae97ecf47" containerID="79e471883aba49654a338815dc971568ffbd1aa6771b7539da8ed80617456316" exitCode=0
Jul 14 00:44:49.804360 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:49.804346    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerDied","Data":"79e471883aba49654a338815dc971568ffbd1aa6771b7539da8ed80617456316"}
Jul 14 00:44:50.810308 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:50.810269    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-4szch" event={"ID":"58c65e1b-6b8e-499e-8009-1b5ae97ecf47","Type":"ContainerStarted","Data":"5c2ff190f325155f6205235a170878debb7205b942ca8deefa2be32fcc222bbc"}
Jul 14 00:44:50.811780 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:50.811749    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-lkgk6" event={"ID":"3d478348-8510-4b2a-bb83-45abe05300d0","Type":"ContainerStarted","Data":"2537f3d2df45890fcaf44bd7638e034e8f8b4152518f510120df52df661297f8"}
Jul 14 00:44:50.836180 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:50.836127    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-4szch" podStartSLOduration=8.632684171 podStartE2EDuration="39.836108017s" totalImagesPullingTime="31.203423846s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:50.834088666 +0000 UTC m=+40.815173352" watchObservedRunningTime="2026-07-14 00:44:50.836108017 +0000 UTC m=+40.817192684"
Jul 14 00:44:50.848836 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:50.848777    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-lkgk6" podStartSLOduration=1.921288259 podStartE2EDuration="5.848760992s" totalImagesPullingTime="3.927472733s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:44:50.848585886 +0000 UTC m=+40.829670547" watchObservedRunningTime="2026-07-14 00:44:50.848760992 +0000 UTC m=+40.829845631"
Jul 14 00:44:53.227773 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:44:53.227732    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:44:53.228148 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:53.227859    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:44:53.228148 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:44:53.227923    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:45:01.227908632 +0000 UTC m=+51.208993272 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:45:01.282044 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:01.281996    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:45:01.282494 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:01.282149    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:45:01.282494 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:01.282214    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:45:17.28219899 +0000 UTC m=+67.263283630 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:45:12.778909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:12.778878    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-k45vj"
Jul 14 00:45:16.190904 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.190846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:45:16.194001 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.193978    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 14 00:45:16.201223 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:16.201200    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 00:45:16.201293 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:16.201279    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:46:20.201262082 +0000 UTC m=+130.182346721 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : secret "metrics-daemon-secret" not found
Jul 14 00:45:16.291330 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.291291    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:45:16.294288 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.294267    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 14 00:45:16.305849 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.305825    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/d727cbd0-53be-43d2-8798-f91cf7d73d0a-original-pull-secret\") pod \"global-pull-secret-syncer-w5xkv\" (UID: \"d727cbd0-53be-43d2-8798-f91cf7d73d0a\") " pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:45:16.392614 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.392570    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:45:16.395333 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.395313    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 14 00:45:16.405669 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.405635    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 14 00:45:16.416601 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.416570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvtw9\" (UniqueName: \"kubernetes.io/projected/c658907e-9fbe-4320-a480-0ef6ffea21ea-kube-api-access-tvtw9\") pod \"network-check-target-jxflr\" (UID: \"c658907e-9fbe-4320-a480-0ef6ffea21ea\") " pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:45:16.530938 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.530846    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w5xkv"
Jul 14 00:45:16.546266 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.546239    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-tbhhd\""
Jul 14 00:45:16.554301 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.554275    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:45:16.665512 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.665464    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w5xkv"]
Jul 14 00:45:16.669945 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:45:16.669919    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd727cbd0_53be_43d2_8798_f91cf7d73d0a.slice/crio-ed1041392a4e220d568ed3f1bbc293316570c7678350bbb40ae63dfa3b226131 WatchSource:0}: Error finding container ed1041392a4e220d568ed3f1bbc293316570c7678350bbb40ae63dfa3b226131: Status 404 returned error can't find the container with id ed1041392a4e220d568ed3f1bbc293316570c7678350bbb40ae63dfa3b226131
Jul 14 00:45:16.682413 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.682382    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-jxflr"]
Jul 14 00:45:16.685941 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:45:16.685921    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc658907e_9fbe_4320_a480_0ef6ffea21ea.slice/crio-277f0ba62eb88bdc4ed54242bb0a5e15353730d7ee25dd51e8b27d17b4b735d6 WatchSource:0}: Error finding container 277f0ba62eb88bdc4ed54242bb0a5e15353730d7ee25dd51e8b27d17b4b735d6: Status 404 returned error can't find the container with id 277f0ba62eb88bdc4ed54242bb0a5e15353730d7ee25dd51e8b27d17b4b735d6
Jul 14 00:45:16.878220 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.878178    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxflr" event={"ID":"c658907e-9fbe-4320-a480-0ef6ffea21ea","Type":"ContainerStarted","Data":"277f0ba62eb88bdc4ed54242bb0a5e15353730d7ee25dd51e8b27d17b4b735d6"}
Jul 14 00:45:16.879173 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:16.879152    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w5xkv" event={"ID":"d727cbd0-53be-43d2-8798-f91cf7d73d0a","Type":"ContainerStarted","Data":"ed1041392a4e220d568ed3f1bbc293316570c7678350bbb40ae63dfa3b226131"}
Jul 14 00:45:17.298487 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:17.298389    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:45:17.299067 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:17.298550    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:45:17.299067 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:17.298620    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:45:49.298599201 +0000 UTC m=+99.279683864 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:45:21.893423 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:21.893384    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w5xkv" event={"ID":"d727cbd0-53be-43d2-8798-f91cf7d73d0a","Type":"ContainerStarted","Data":"11f3ed25ab0b13edc643759c0dc8824d6da184b744d558d2d882f2715c8daa69"}
Jul 14 00:45:21.894788 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:21.894768    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-jxflr" event={"ID":"c658907e-9fbe-4320-a480-0ef6ffea21ea","Type":"ContainerStarted","Data":"974f92d6153b633d290a7a4523356b0acd4cce46fe5db377f0f3ab828743dc28"}
Jul 14 00:45:21.894902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:21.894890    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:45:21.908141 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:21.908090    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w5xkv" podStartSLOduration=66.423616288 podStartE2EDuration="1m10.908075227s" totalImagesPullingTime="4.484458939s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:45:21.907592117 +0000 UTC m=+71.888676778" watchObservedRunningTime="2026-07-14 00:45:21.908075227 +0000 UTC m=+71.889159941"
Jul 14 00:45:21.922602 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:21.922548    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-jxflr" podStartSLOduration=66.456959206 podStartE2EDuration="1m10.922531608s" totalImagesPullingTime="4.465572402s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:45:21.921881228 +0000 UTC m=+71.902965893" watchObservedRunningTime="2026-07-14 00:45:21.922531608 +0000 UTC m=+71.903616249"
Jul 14 00:45:44.230029 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.229895    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"]
Jul 14 00:45:44.233634 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.233485    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.236536 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.236517    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 14 00:45:44.236649 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.236555    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 14 00:45:44.237709 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.237680    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 14 00:45:44.237809 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.237718    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 14 00:45:44.237809 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.237734    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 14 00:45:44.237809 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.237745    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 14 00:45:44.237809 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.237734    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 14 00:45:44.243664 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.243640    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"]
Jul 14 00:45:44.284125 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.284085    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.284125 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.284125    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-hub\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.284342 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.284149    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.284342 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.284258    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-ca\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.284342 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.284285    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tmtjl\" (UniqueName: \"kubernetes.io/projected/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-kube-api-access-tmtjl\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.284342 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.284313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.385437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.385396    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.385437 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.385440    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-hub\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.385704 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.385470    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.385704 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.385545    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-ca\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.385704 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.385570    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tmtjl\" (UniqueName: \"kubernetes.io/projected/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-kube-api-access-tmtjl\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.385704 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.385595    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.386269 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.386240    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-additional-service-ca\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.388033 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.388010    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-ca\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.388166 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.388147    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-hub\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.388365 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.388350    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.388400 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.388379    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.394576 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.394550    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tmtjl\" (UniqueName: \"kubernetes.io/projected/c26f12f8-a77a-4156-b3e4-6fe8c6cccb34-kube-api-access-tmtjl\") pod \"cluster-proxy-proxy-agent-7498d897dc-rvfzt\" (UID: \"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.553545 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.553419    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:45:44.679678 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.679645    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"]
Jul 14 00:45:44.682744 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:45:44.682721    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc26f12f8_a77a_4156_b3e4_6fe8c6cccb34.slice/crio-67f010b33c33b83a4b08bba13f903bbf7546236103ce2a950222788416c188ae WatchSource:0}: Error finding container 67f010b33c33b83a4b08bba13f903bbf7546236103ce2a950222788416c188ae: Status 404 returned error can't find the container with id 67f010b33c33b83a4b08bba13f903bbf7546236103ce2a950222788416c188ae
Jul 14 00:45:44.949570 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:44.949536    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" event={"ID":"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34","Type":"ContainerStarted","Data":"67f010b33c33b83a4b08bba13f903bbf7546236103ce2a950222788416c188ae"}
Jul 14 00:45:48.962660 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:48.962625    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" event={"ID":"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34","Type":"ContainerStarted","Data":"a2b7eefa3157786511bb084390e73fce7aaf9ad85ba492c19e0667a67675deff"}
Jul 14 00:45:48.963073 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:48.962669    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" event={"ID":"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34","Type":"ContainerStarted","Data":"2fa540548cf2554c6e0f1d674bbfdf55463fe3fa4cbbbc004a4b8706ed1d1d33"}
Jul 14 00:45:48.963073 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:48.962688    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" event={"ID":"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34","Type":"ContainerStarted","Data":"164dea579702fd701bf0aab3841167f85a7d00257f69a7210bb936f955c8408d"}
Jul 14 00:45:48.981403 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:48.981345    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" podStartSLOduration=1.297650757 podStartE2EDuration="4.981325201s" totalImagesPullingTime="3.683674444s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:45:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:45:48.981080284 +0000 UTC m=+98.962164945" watchObservedRunningTime="2026-07-14 00:45:48.981325201 +0000 UTC m=+98.962409861"
Jul 14 00:45:49.327086 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:49.327048    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:45:49.327236 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:49.327198    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:45:49.327279 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:45:49.327262    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:46:53.327247403 +0000 UTC m=+163.308332042 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:45:52.899560 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:45:52.899525    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-jxflr"
Jul 14 00:46:20.242351 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:46:20.242297    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:46:20.242846 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:46:20.242466    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 00:46:20.242846 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:46:20.242570    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:22.242552457 +0000 UTC m=+252.223637096 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : secret "metrics-daemon-secret" not found
Jul 14 00:46:48.425627 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:46:48.425576    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qqm2z" podUID="7cdd529a-053a-4f90-8e1f-88143f580ff9"
Jul 14 00:46:49.130159 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:46:49.130127    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:46:49.537907 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:46:49.537804    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:46:53.375463 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:46:53.375415    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:46:53.375914 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:46:53.375569    2552 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 14 00:46:53.375914 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:46:53.375631    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls podName:7cdd529a-053a-4f90-8e1f-88143f580ff9 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:55.375617662 +0000 UTC m=+285.356702303 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls") pod "dns-default-qqm2z" (UID: "7cdd529a-053a-4f90-8e1f-88143f580ff9") : secret "dns-default-metrics-tls" not found
Jul 14 00:47:01.519177 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:47:01.519129    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:48:12.713667 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.713632    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"]
Jul 14 00:48:12.716321 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.716305    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"
Jul 14 00:48:12.718864 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.718835    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-storage-operator\"/\"volume-data-source-validator-dockercfg-smlkk\""
Jul 14 00:48:12.720050 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.720023    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"kube-root-ca.crt\""
Jul 14 00:48:12.720050 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.720042    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-storage-operator\"/\"openshift-service-ca.crt\""
Jul 14 00:48:12.727445 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.727417    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"]
Jul 14 00:48:12.812761 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.812726    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rncw\" (UniqueName: \"kubernetes.io/projected/fd5d08e6-b26e-4d6b-b21d-f9d23ed05783-kube-api-access-2rncw\") pod \"volume-data-source-validator-78495597f6-p5ltk\" (UID: \"fd5d08e6-b26e-4d6b-b21d-f9d23ed05783\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"
Jul 14 00:48:12.913834 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.913799    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rncw\" (UniqueName: \"kubernetes.io/projected/fd5d08e6-b26e-4d6b-b21d-f9d23ed05783-kube-api-access-2rncw\") pod \"volume-data-source-validator-78495597f6-p5ltk\" (UID: \"fd5d08e6-b26e-4d6b-b21d-f9d23ed05783\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"
Jul 14 00:48:12.921464 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:12.921437    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rncw\" (UniqueName: \"kubernetes.io/projected/fd5d08e6-b26e-4d6b-b21d-f9d23ed05783-kube-api-access-2rncw\") pod \"volume-data-source-validator-78495597f6-p5ltk\" (UID: \"fd5d08e6-b26e-4d6b-b21d-f9d23ed05783\") " pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"
Jul 14 00:48:13.025069 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:13.024959    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"
Jul 14 00:48:13.146657 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:13.146620    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk"]
Jul 14 00:48:13.149850 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:13.149819    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd5d08e6_b26e_4d6b_b21d_f9d23ed05783.slice/crio-52ca3bca061d580fc77ecb10f4b4870e6d38137022b60493388cedcb1f43d0f0 WatchSource:0}: Error finding container 52ca3bca061d580fc77ecb10f4b4870e6d38137022b60493388cedcb1f43d0f0: Status 404 returned error can't find the container with id 52ca3bca061d580fc77ecb10f4b4870e6d38137022b60493388cedcb1f43d0f0
Jul 14 00:48:13.363594 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:13.363559    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk" event={"ID":"fd5d08e6-b26e-4d6b-b21d-f9d23ed05783","Type":"ContainerStarted","Data":"52ca3bca061d580fc77ecb10f4b4870e6d38137022b60493388cedcb1f43d0f0"}
Jul 14 00:48:15.370902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:15.370865    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk" event={"ID":"fd5d08e6-b26e-4d6b-b21d-f9d23ed05783","Type":"ContainerStarted","Data":"0ea91d134c24816f34ca887a2163c73d6cd2dc61bbeab43b4d8aea6ad25bfd59"}
Jul 14 00:48:15.384685 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:15.384622    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-storage-operator/volume-data-source-validator-78495597f6-p5ltk" podStartSLOduration=2.082863195 podStartE2EDuration="3.384601381s" totalImagesPullingTime="1.301738186s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:48:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:48:15.384558148 +0000 UTC m=+245.365642811" watchObservedRunningTime="2026-07-14 00:48:15.384601381 +0000 UTC m=+245.365686042"
Jul 14 00:48:16.974999 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.974963    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-qsrmk"]
Jul 14 00:48:16.977966 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.977948    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:16.980732 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.980706    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 14 00:48:16.980732 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.980720    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 14 00:48:16.980903 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.980736    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 14 00:48:16.981806 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.981789    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 14 00:48:16.981897 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.981809    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-942xj\""
Jul 14 00:48:16.986849 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:16.986825    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qsrmk"]
Jul 14 00:48:17.147787 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.147737    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ncnf9\" (UniqueName: \"kubernetes.io/projected/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-kube-api-access-ncnf9\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.147787 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.147797    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-crio-socket\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.148001 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.147827    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.148001 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.147858    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-data-volume\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.148001 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.147874    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.248797 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.248703    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-data-volume\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.248797 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.248743    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.248797 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.248774    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ncnf9\" (UniqueName: \"kubernetes.io/projected/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-kube-api-access-ncnf9\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.249045 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.248805    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-crio-socket\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.249045 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.248834    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.249045 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.248936    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-crio-socket\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.249045 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:17.248979    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:17.249222 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:17.249055    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls podName:7629acf0-e2b5-406d-ba54-d1dca19b1bc9 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:17.749034797 +0000 UTC m=+247.730119452 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsrmk" (UID: "7629acf0-e2b5-406d-ba54-d1dca19b1bc9") : secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:17.249222 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.249177    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-data-volume\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.249368 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.249351    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.257027 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.257001    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ncnf9\" (UniqueName: \"kubernetes.io/projected/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-kube-api-access-ncnf9\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.753349 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:17.753299    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:17.753608 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:17.753448    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:17.753608 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:17.753531    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls podName:7629acf0-e2b5-406d-ba54-d1dca19b1bc9 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:18.75351481 +0000 UTC m=+248.734599463 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsrmk" (UID: "7629acf0-e2b5-406d-ba54-d1dca19b1bc9") : secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:18.761382 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.761338    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:18.761886 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:18.761464    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:18.761886 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:18.761546    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls podName:7629acf0-e2b5-406d-ba54-d1dca19b1bc9 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:20.761530344 +0000 UTC m=+250.742614985 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsrmk" (UID: "7629acf0-e2b5-406d-ba54-d1dca19b1bc9") : secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:18.784012 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.783981    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"]
Jul 14 00:48:18.787055 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.787038    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"
Jul 14 00:48:18.789715 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.789695    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Jul 14 00:48:18.789833 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.789694    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Jul 14 00:48:18.790824 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.790808    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-blsrn\""
Jul 14 00:48:18.795215 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.795191    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"]
Jul 14 00:48:18.953998 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.953975    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gbfs8_44023eff-43a4-4016-a62c-f10e986d2a18/dns-node-resolver/0.log"
Jul 14 00:48:18.962246 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:18.962210    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsdvd\" (UniqueName: \"kubernetes.io/projected/8e04aff9-8c81-4a11-9cd2-64c312934cff-kube-api-access-jsdvd\") pod \"migrator-7f59b54f6c-b49lh\" (UID: \"8e04aff9-8c81-4a11-9cd2-64c312934cff\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"
Jul 14 00:48:19.063581 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:19.063479    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsdvd\" (UniqueName: \"kubernetes.io/projected/8e04aff9-8c81-4a11-9cd2-64c312934cff-kube-api-access-jsdvd\") pod \"migrator-7f59b54f6c-b49lh\" (UID: \"8e04aff9-8c81-4a11-9cd2-64c312934cff\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"
Jul 14 00:48:19.071775 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:19.071741    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsdvd\" (UniqueName: \"kubernetes.io/projected/8e04aff9-8c81-4a11-9cd2-64c312934cff-kube-api-access-jsdvd\") pod \"migrator-7f59b54f6c-b49lh\" (UID: \"8e04aff9-8c81-4a11-9cd2-64c312934cff\") " pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"
Jul 14 00:48:19.096715 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:19.096683    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"
Jul 14 00:48:19.225107 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:19.225073    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh"]
Jul 14 00:48:19.228431 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:19.228397    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e04aff9_8c81_4a11_9cd2_64c312934cff.slice/crio-7408cab2ced724b50cdcb9a78a828097edb9bb67feca7014ac0d086f0c018950 WatchSource:0}: Error finding container 7408cab2ced724b50cdcb9a78a828097edb9bb67feca7014ac0d086f0c018950: Status 404 returned error can't find the container with id 7408cab2ced724b50cdcb9a78a828097edb9bb67feca7014ac0d086f0c018950
Jul 14 00:48:19.384914 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:19.384879    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh" event={"ID":"8e04aff9-8c81-4a11-9cd2-64c312934cff","Type":"ContainerStarted","Data":"7408cab2ced724b50cdcb9a78a828097edb9bb67feca7014ac0d086f0c018950"}
Jul 14 00:48:20.153702 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:20.153669    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wq6ck_a13d6513-ea94-485d-adf6-532636f8112c/node-ca/0.log"
Jul 14 00:48:20.755482 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:20.755452    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lkgk6_3d478348-8510-4b2a-bb83-45abe05300d0/serve-healthcheck-canary/0.log"
Jul 14 00:48:20.778665 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:20.778622    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:20.778848 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:20.778830    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:20.778917 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:20.778906    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls podName:7629acf0-e2b5-406d-ba54-d1dca19b1bc9 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:24.778884331 +0000 UTC m=+254.759968977 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsrmk" (UID: "7629acf0-e2b5-406d-ba54-d1dca19b1bc9") : secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:21.394571 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:21.394534    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh" event={"ID":"8e04aff9-8c81-4a11-9cd2-64c312934cff","Type":"ContainerStarted","Data":"4a4e66e3e04ce724975cacd33876121ae5d11de74284ab72617910f6512e57f7"}
Jul 14 00:48:21.394970 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:21.394621    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh" event={"ID":"8e04aff9-8c81-4a11-9cd2-64c312934cff","Type":"ContainerStarted","Data":"34c39e035196792f8bd2df1677c9f5754199eb0e457cf0caba3dc4e79feb6152"}
Jul 14 00:48:21.410966 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:21.410910    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-7f59b54f6c-b49lh" podStartSLOduration=1.9704135919999999 podStartE2EDuration="3.410893335s" totalImagesPullingTime="1.440479743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:48:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:48:21.410874374 +0000 UTC m=+251.391959037" watchObservedRunningTime="2026-07-14 00:48:21.410893335 +0000 UTC m=+251.391977996"
Jul 14 00:48:22.287180 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:22.287135    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:48:22.287387 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:22.287261    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Jul 14 00:48:22.287387 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:22.287329    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs podName:79b4894c-cac8-408b-ae28-e3c6788ba884 nodeName:}" failed. No retries permitted until 2026-07-14 00:50:24.287310411 +0000 UTC m=+374.268395070 (durationBeforeRetry 2m2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs") pod "network-metrics-daemon-7bcw7" (UID: "79b4894c-cac8-408b-ae28-e3c6788ba884") : secret "metrics-daemon-secret" not found
Jul 14 00:48:24.805762 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:24.805722    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:24.806264 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:24.805853    2552 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:24.806264 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:24.805937    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls podName:7629acf0-e2b5-406d-ba54-d1dca19b1bc9 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:32.805914536 +0000 UTC m=+262.786999198 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls") pod "insights-runtime-extractor-qsrmk" (UID: "7629acf0-e2b5-406d-ba54-d1dca19b1bc9") : secret "insights-runtime-extractor-tls" not found
Jul 14 00:48:32.857135 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:32.857045    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:32.859295 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:32.859271    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/7629acf0-e2b5-406d-ba54-d1dca19b1bc9-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-qsrmk\" (UID: \"7629acf0-e2b5-406d-ba54-d1dca19b1bc9\") " pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:32.888281 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:32.888224    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-qsrmk"
Jul 14 00:48:33.022450 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:33.022413    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-qsrmk"]
Jul 14 00:48:33.026572 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:33.026545    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7629acf0_e2b5_406d_ba54_d1dca19b1bc9.slice/crio-d26d5932ad907296e6a107835b8d16a6aeebfcd5302faf19dc81013a399a4c10 WatchSource:0}: Error finding container d26d5932ad907296e6a107835b8d16a6aeebfcd5302faf19dc81013a399a4c10: Status 404 returned error can't find the container with id d26d5932ad907296e6a107835b8d16a6aeebfcd5302faf19dc81013a399a4c10
Jul 14 00:48:33.431982 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:33.431944    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsrmk" event={"ID":"7629acf0-e2b5-406d-ba54-d1dca19b1bc9","Type":"ContainerStarted","Data":"d26d5932ad907296e6a107835b8d16a6aeebfcd5302faf19dc81013a399a4c10"}
Jul 14 00:48:37.447635 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:37.447595    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsrmk" event={"ID":"7629acf0-e2b5-406d-ba54-d1dca19b1bc9","Type":"ContainerStarted","Data":"78be55536dd337e1315a8e5381460a8bd3b115c0aad57b8f89cda64c445dacf3"}
Jul 14 00:48:38.453772 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:38.453731    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsrmk" event={"ID":"7629acf0-e2b5-406d-ba54-d1dca19b1bc9","Type":"ContainerStarted","Data":"1c8f666a21be6f21783ec1ec15a749b18267e2866a93fc38c57e7ae340b70d15"}
Jul 14 00:48:40.462625 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:40.462585    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-qsrmk" event={"ID":"7629acf0-e2b5-406d-ba54-d1dca19b1bc9","Type":"ContainerStarted","Data":"d04c8b1d4843a17a52978c7f1298572be6efdf49e5e9470ac974660f7adfc9bd"}
Jul 14 00:48:40.480764 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:40.480715    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-qsrmk" podStartSLOduration=17.555092369 podStartE2EDuration="24.480699363s" totalImagesPullingTime="6.925606994s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 00:48:16 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:48:40.479385326 +0000 UTC m=+270.460470031" watchObservedRunningTime="2026-07-14 00:48:40.480699363 +0000 UTC m=+270.461784022"
Jul 14 00:48:47.332776 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.332738    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"]
Jul 14 00:48:47.335902 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.335882    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:47.338509 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.338475    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Jul 14 00:48:47.338509 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.338507    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-tt4ps\""
Jul 14 00:48:47.343472 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.343453    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"]
Jul 14 00:48:47.359869 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.359830    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2da1d0-434f-4acd-9435-1958b73fc513-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-z7sp5\" (UID: \"cf2da1d0-434f-4acd-9435-1958b73fc513\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:47.460232 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.460196    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2da1d0-434f-4acd-9435-1958b73fc513-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-z7sp5\" (UID: \"cf2da1d0-434f-4acd-9435-1958b73fc513\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:47.460354 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:47.460313    2552 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-admission-webhook-tls: secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 00:48:47.460395 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:47.460369    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/cf2da1d0-434f-4acd-9435-1958b73fc513-tls-certificates podName:cf2da1d0-434f-4acd-9435-1958b73fc513 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:47.960354689 +0000 UTC m=+277.941439329 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certificates" (UniqueName: "kubernetes.io/secret/cf2da1d0-434f-4acd-9435-1958b73fc513-tls-certificates") pod "prometheus-operator-admission-webhook-77d98bc786-z7sp5" (UID: "cf2da1d0-434f-4acd-9435-1958b73fc513") : secret "prometheus-operator-admission-webhook-tls" not found
Jul 14 00:48:47.964650 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.964597    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2da1d0-434f-4acd-9435-1958b73fc513-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-z7sp5\" (UID: \"cf2da1d0-434f-4acd-9435-1958b73fc513\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:47.967015 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:47.966985    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cf2da1d0-434f-4acd-9435-1958b73fc513-tls-certificates\") pod \"prometheus-operator-admission-webhook-77d98bc786-z7sp5\" (UID: \"cf2da1d0-434f-4acd-9435-1958b73fc513\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:48.245146 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:48.245054    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:48.369457 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:48.369424    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"]
Jul 14 00:48:48.373035 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:48.373009    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf2da1d0_434f_4acd_9435_1958b73fc513.slice/crio-d0789f77bc7c743ac16369b9b7ec5ed5d97c34d7c0f29744cdcfd558136d8076 WatchSource:0}: Error finding container d0789f77bc7c743ac16369b9b7ec5ed5d97c34d7c0f29744cdcfd558136d8076: Status 404 returned error can't find the container with id d0789f77bc7c743ac16369b9b7ec5ed5d97c34d7c0f29744cdcfd558136d8076
Jul 14 00:48:48.491450 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:48.491412    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5" event={"ID":"cf2da1d0-434f-4acd-9435-1958b73fc513","Type":"ContainerStarted","Data":"d0789f77bc7c743ac16369b9b7ec5ed5d97c34d7c0f29744cdcfd558136d8076"}
Jul 14 00:48:50.498255 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:50.498223    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5" event={"ID":"cf2da1d0-434f-4acd-9435-1958b73fc513","Type":"ContainerStarted","Data":"49d0e792d1aa610d1309a2bbb657b97363b7b08a86fe9e528448f1250c127ad4"}
Jul 14 00:48:50.498843 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:50.498825    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:50.504648 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:50.504622    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5"
Jul 14 00:48:50.525282 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:50.525201    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-77d98bc786-z7sp5" podStartSLOduration=2.380725561 podStartE2EDuration="3.525185232s" totalImagesPullingTime="1.144459671s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:48:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:48:50.524465405 +0000 UTC m=+280.505550068" watchObservedRunningTime="2026-07-14 00:48:50.525185232 +0000 UTC m=+280.506269893"
Jul 14 00:48:51.393579 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.393543    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-nhsd4"]
Jul 14 00:48:51.396563 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.396545    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.399417 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.399393    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Jul 14 00:48:51.400306 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.400283    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 14 00:48:51.400306 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.400295    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 14 00:48:51.400486 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.400295    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 14 00:48:51.400486 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.400290    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Jul 14 00:48:51.400768 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.400751    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-5nkqp\""
Jul 14 00:48:51.404652 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.404626    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-nhsd4"]
Jul 14 00:48:51.491047 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.491000    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.491047 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.491042    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqsxc\" (UniqueName: \"kubernetes.io/projected/aa536561-ee97-4596-8730-24c79819ef86-kube-api-access-zqsxc\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.491278 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.491062    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.491278 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.491108    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa536561-ee97-4596-8730-24c79819ef86-metrics-client-ca\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.591672 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.591633    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.591672 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.591675    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqsxc\" (UniqueName: \"kubernetes.io/projected/aa536561-ee97-4596-8730-24c79819ef86-kube-api-access-zqsxc\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.592205 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.591705    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.592205 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:51.591791    2552 secret.go:189] Couldn't get secret openshift-monitoring/prometheus-operator-tls: secret "prometheus-operator-tls" not found
Jul 14 00:48:51.592205 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:51.591880    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-tls podName:aa536561-ee97-4596-8730-24c79819ef86 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:52.091860358 +0000 UTC m=+282.072945001 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "prometheus-operator-tls" (UniqueName: "kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-tls") pod "prometheus-operator-64785b6675-nhsd4" (UID: "aa536561-ee97-4596-8730-24c79819ef86") : secret "prometheus-operator-tls" not found
Jul 14 00:48:51.592205 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.591899    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa536561-ee97-4596-8730-24c79819ef86-metrics-client-ca\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.592601 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.592580    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/aa536561-ee97-4596-8730-24c79819ef86-metrics-client-ca\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.593967 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.593948    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:51.602510 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:51.602475    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqsxc\" (UniqueName: \"kubernetes.io/projected/aa536561-ee97-4596-8730-24c79819ef86-kube-api-access-zqsxc\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:52.095349 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:52.095306    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:52.097795 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:52.097772    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/aa536561-ee97-4596-8730-24c79819ef86-prometheus-operator-tls\") pod \"prometheus-operator-64785b6675-nhsd4\" (UID: \"aa536561-ee97-4596-8730-24c79819ef86\") " pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:52.130777 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:52.130716    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-qqm2z" podUID="7cdd529a-053a-4f90-8e1f-88143f580ff9"
Jul 14 00:48:52.306632 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:52.306583    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4"
Jul 14 00:48:52.430670 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:52.430637    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-64785b6675-nhsd4"]
Jul 14 00:48:52.433711 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:52.433689    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa536561_ee97_4596_8730_24c79819ef86.slice/crio-49f09033636bc2d9c0485d9bd0cc0dc0affc0b40994c4c5be4025a785b2dd641 WatchSource:0}: Error finding container 49f09033636bc2d9c0485d9bd0cc0dc0affc0b40994c4c5be4025a785b2dd641: Status 404 returned error can't find the container with id 49f09033636bc2d9c0485d9bd0cc0dc0affc0b40994c4c5be4025a785b2dd641
Jul 14 00:48:52.505542 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:52.505494    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:48:52.505542 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:52.505523    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4" event={"ID":"aa536561-ee97-4596-8730-24c79819ef86","Type":"ContainerStarted","Data":"49f09033636bc2d9c0485d9bd0cc0dc0affc0b40994c4c5be4025a785b2dd641"}
Jul 14 00:48:54.514443 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:54.514360    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4" event={"ID":"aa536561-ee97-4596-8730-24c79819ef86","Type":"ContainerStarted","Data":"b0329f8ca3eb04eaba693369f42db52843f30be19d1bec2ea222c39b9d1c2c2f"}
Jul 14 00:48:54.514443 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:54.514409    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4" event={"ID":"aa536561-ee97-4596-8730-24c79819ef86","Type":"ContainerStarted","Data":"27ed384801ee3d601f621fbd80e76f0e94a76f85a1e18376f9638adba875ebc5"}
Jul 14 00:48:54.531303 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:54.531248    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-64785b6675-nhsd4" podStartSLOduration=1.7697530110000002 podStartE2EDuration="3.531232724s" totalImagesPullingTime="1.761479713s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:48:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:48:54.530920949 +0000 UTC m=+284.512005628" watchObservedRunningTime="2026-07-14 00:48:54.531232724 +0000 UTC m=+284.512317385"
Jul 14 00:48:55.419408 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:55.419365    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:48:55.421882 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:55.421847    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7cdd529a-053a-4f90-8e1f-88143f580ff9-metrics-tls\") pod \"dns-default-qqm2z\" (UID: \"7cdd529a-053a-4f90-8e1f-88143f580ff9\") " pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:48:55.509185 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:55.509155    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-rvlnc\""
Jul 14 00:48:55.516932 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:55.516900    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:48:55.645844 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:55.645810    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-qqm2z"]
Jul 14 00:48:55.648861 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:55.648825    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7cdd529a_053a_4f90_8e1f_88143f580ff9.slice/crio-db951e0ce6e31b8f574c222fd04169ea1aae75cca99ac335442e44cfcedb750d WatchSource:0}: Error finding container db951e0ce6e31b8f574c222fd04169ea1aae75cca99ac335442e44cfcedb750d: Status 404 returned error can't find the container with id db951e0ce6e31b8f574c222fd04169ea1aae75cca99ac335442e44cfcedb750d
Jul 14 00:48:56.523489 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.523450    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqm2z" event={"ID":"7cdd529a-053a-4f90-8e1f-88143f580ff9","Type":"ContainerStarted","Data":"db951e0ce6e31b8f574c222fd04169ea1aae75cca99ac335442e44cfcedb750d"}
Jul 14 00:48:56.741143 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.741106    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-p4trm"]
Jul 14 00:48:56.744809 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.744781    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.747969 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.747946    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 14 00:48:56.748130 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.747998    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 14 00:48:56.748130 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.747998    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6nrc5\""
Jul 14 00:48:56.748227 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.748142    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 14 00:48:56.830672 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830645    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-root\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.830832 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830676    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-wtmp\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.830832 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830708    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-accelerators-collector-config\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.830832 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830778    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-tls\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.830832 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830796    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-sys\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.830832 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830822    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752e8910-0a6b-46b0-bc78-819374a99f62-metrics-client-ca\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.831009 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830836    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.831009 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830957    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9c8k\" (UniqueName: \"kubernetes.io/projected/752e8910-0a6b-46b0-bc78-819374a99f62-kube-api-access-p9c8k\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.831009 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.830992    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-textfile\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.931861 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.931817    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-tls\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.931861 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.931869    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-sys\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.931902    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752e8910-0a6b-46b0-bc78-819374a99f62-metrics-client-ca\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.931927    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.931976    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-sys\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.931993    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9c8k\" (UniqueName: \"kubernetes.io/projected/752e8910-0a6b-46b0-bc78-819374a99f62-kube-api-access-p9c8k\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932120 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932050    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-textfile\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932348 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932203    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-root\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932348 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932232    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-wtmp\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932348 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932257    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-accelerators-collector-config\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932487 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932348    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-root\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932585 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932488    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-textfile\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932585 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932517    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-wtmp\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932858 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932801    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/752e8910-0a6b-46b0-bc78-819374a99f62-metrics-client-ca\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.932977 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.932955    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-accelerators-collector-config\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.934706 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.934680    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-tls\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.934836 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.934814    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/752e8910-0a6b-46b0-bc78-819374a99f62-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:56.941406 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:56.941374    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9c8k\" (UniqueName: \"kubernetes.io/projected/752e8910-0a6b-46b0-bc78-819374a99f62-kube-api-access-p9c8k\") pod \"node-exporter-p4trm\" (UID: \"752e8910-0a6b-46b0-bc78-819374a99f62\") " pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:57.055376 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.055335    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-p4trm"
Jul 14 00:48:57.251146 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:57.251114    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod752e8910_0a6b_46b0_bc78_819374a99f62.slice/crio-c15cfeee991f05001a4c8099fa2952843944d0d2e79a5f60661eed4b07253374 WatchSource:0}: Error finding container c15cfeee991f05001a4c8099fa2952843944d0d2e79a5f60661eed4b07253374: Status 404 returned error can't find the container with id c15cfeee991f05001a4c8099fa2952843944d0d2e79a5f60661eed4b07253374
Jul 14 00:48:57.529029 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.528900    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqm2z" event={"ID":"7cdd529a-053a-4f90-8e1f-88143f580ff9","Type":"ContainerStarted","Data":"68c81bcb8b5cd84cd8999a81ab906e44e0c194e4db5e869e2126828adc5aaa25"}
Jul 14 00:48:57.529029 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.528971    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-qqm2z" event={"ID":"7cdd529a-053a-4f90-8e1f-88143f580ff9","Type":"ContainerStarted","Data":"eac20b66a94e0a68ef3b1340f3be31ca4561ddb2b5037babb514218a20f453b6"}
Jul 14 00:48:57.529631 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.529040    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:48:57.530217 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.530196    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p4trm" event={"ID":"752e8910-0a6b-46b0-bc78-819374a99f62","Type":"ContainerStarted","Data":"c15cfeee991f05001a4c8099fa2952843944d0d2e79a5f60661eed4b07253374"}
Jul 14 00:48:57.546681 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.546618    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-qqm2z" podStartSLOduration=250.903322539 podStartE2EDuration="4m12.546594952s" totalImagesPullingTime="1.643272413s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:48:57.54584096 +0000 UTC m=+287.526925621" watchObservedRunningTime="2026-07-14 00:48:57.546594952 +0000 UTC m=+287.527679614"
Jul 14 00:48:57.812302 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.812209    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 00:48:57.816040 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.816011    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.818659 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.818611    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Jul 14 00:48:57.818659 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.818613    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Jul 14 00:48:57.818914 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.818765    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Jul 14 00:48:57.818914 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.818776    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Jul 14 00:48:57.818914 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.818819    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Jul 14 00:48:57.819100 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.819032    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Jul 14 00:48:57.819100 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.819043    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Jul 14 00:48:57.819100 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.819046    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-6mc95\""
Jul 14 00:48:57.819100 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.819093    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Jul 14 00:48:57.819350 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.819335    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Jul 14 00:48:57.832391 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.832350    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 00:48:57.842042 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842010    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842227 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842054    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842227 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-config-out\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842227 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lj5z\" (UniqueName: \"kubernetes.io/projected/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-kube-api-access-6lj5z\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842227 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842164    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842227 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842188    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-web-config\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842561 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842241    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842561 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842267    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842561 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842333    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842561 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842395    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842561 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842434    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842561 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842463    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.842561 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.842490    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.942928 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.942887    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943070 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.942942    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943070 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.942967    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943140 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:48:57.943084    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-trusted-ca-bundle podName:f866da78-6c90-4a3a-83cd-3afed9cfd0a9 nodeName:}" failed. No retries permitted until 2026-07-14 00:48:58.443064604 +0000 UTC m=+288.424149243 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "f866da78-6c90-4a3a-83cd-3afed9cfd0a9") : configmap references non-existent config key: ca-bundle.crt
Jul 14 00:48:57.943221 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.943139    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943221 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.943177    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943221 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.943209    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943346 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.943238    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943346 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.943264    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-config-out\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943346 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.943288    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lj5z\" (UniqueName: \"kubernetes.io/projected/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-kube-api-access-6lj5z\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.943346 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.943312    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.944244 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.944124    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-web-config\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.944244 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.944221    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.944442 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.944259    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.948552 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.946279    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.948552 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.946832    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.948552 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.947342    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.948552 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.947533    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.948552 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.948425    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-config-out\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.948914 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.948764    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-tls-assets\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.951525 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.951354    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-web-config\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.951648 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.951541    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.953694 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.953668    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-config-volume\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.953816 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.953777    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.954834 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.954811    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:57.960729 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:57.960696    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lj5z\" (UniqueName: \"kubernetes.io/projected/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-kube-api-access-6lj5z\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:58.448251 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:58.448146    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:58.448975 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:58.448952    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/f866da78-6c90-4a3a-83cd-3afed9cfd0a9-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"f866da78-6c90-4a3a-83cd-3afed9cfd0a9\") " pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:58.534576 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:58.534538    2552 generic.go:356] "Generic (PLEG): container finished" podID="752e8910-0a6b-46b0-bc78-819374a99f62" containerID="8f229a23ed0b3c9341cd7f09176b31ae761bd381580b0dff8f3cac25a1f493f8" exitCode=0
Jul 14 00:48:58.534913 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:58.534620    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p4trm" event={"ID":"752e8910-0a6b-46b0-bc78-819374a99f62","Type":"ContainerDied","Data":"8f229a23ed0b3c9341cd7f09176b31ae761bd381580b0dff8f3cac25a1f493f8"}
Jul 14 00:48:58.727639 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:58.727555    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Jul 14 00:48:58.867710 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:58.867674    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Jul 14 00:48:58.871269 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:48:58.871242    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf866da78_6c90_4a3a_83cd_3afed9cfd0a9.slice/crio-506f71e4acec1e636d55c20c367910a5b8def7744f50c55040f5f1657a766d26 WatchSource:0}: Error finding container 506f71e4acec1e636d55c20c367910a5b8def7744f50c55040f5f1657a766d26: Status 404 returned error can't find the container with id 506f71e4acec1e636d55c20c367910a5b8def7744f50c55040f5f1657a766d26
Jul 14 00:48:59.539871 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:59.539834    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p4trm" event={"ID":"752e8910-0a6b-46b0-bc78-819374a99f62","Type":"ContainerStarted","Data":"df33a93bcafeab11ca51c42f66ac38d050db8e9d67bdf5aeb7fbf1dfc73ecb8a"}
Jul 14 00:48:59.540268 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:59.539894    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-p4trm" event={"ID":"752e8910-0a6b-46b0-bc78-819374a99f62","Type":"ContainerStarted","Data":"1e15d2b3dc777acc6acf2c550bb4fa4ca8102b30420ba7904e669a528845521e"}
Jul 14 00:48:59.540879 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:59.540857    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerStarted","Data":"506f71e4acec1e636d55c20c367910a5b8def7744f50c55040f5f1657a766d26"}
Jul 14 00:48:59.562615 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:48:59.562559    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-p4trm" podStartSLOduration=2.600199434 podStartE2EDuration="3.562541161s" totalImagesPullingTime="897.472287ms" totalInitContainerRuntime="64.86944ms" isStatefulPod=true podCreationTimestamp="2026-07-14 00:48:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:48:59.561592991 +0000 UTC m=+289.542677652" watchObservedRunningTime="2026-07-14 00:48:59.562541161 +0000 UTC m=+289.543625822"
Jul 14 00:49:00.545763 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:00.545672    2552 generic.go:356] "Generic (PLEG): container finished" podID="f866da78-6c90-4a3a-83cd-3afed9cfd0a9" containerID="450c8892ac325fc17b1ed5bf18982e0822d96fa28f0c6500b89038ff51f3cf03" exitCode=0
Jul 14 00:49:00.546146 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:00.545770    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerDied","Data":"450c8892ac325fc17b1ed5bf18982e0822d96fa28f0c6500b89038ff51f3cf03"}
Jul 14 00:49:02.987196 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.987146    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 00:49:02.990346 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.990314    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:02.992985 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.992949    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Jul 14 00:49:02.993123 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.993018    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Jul 14 00:49:02.993310 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.993292    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Jul 14 00:49:02.993357 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.993342    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Jul 14 00:49:02.993467 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.993292    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Jul 14 00:49:02.994070 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.994042    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Jul 14 00:49:02.994328 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.994310    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Jul 14 00:49:02.994422 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.994326    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Jul 14 00:49:02.994422 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.994357    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Jul 14 00:49:02.994572 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.994549    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Jul 14 00:49:02.995168 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.995139    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Jul 14 00:49:02.995168 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.995149    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Jul 14 00:49:02.995168 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.995142    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-nz8x2\""
Jul 14 00:49:02.995518 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.995477    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Jul 14 00:49:02.995731 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.995709    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-3fffrm1466ved\""
Jul 14 00:49:02.996795 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:02.996775    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Jul 14 00:49:03.000689 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.000667    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Jul 14 00:49:03.011826 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.011771    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 00:49:03.089651 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089568    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089651 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089607    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089651 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089625    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089651 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089646    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089663    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-config\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089692    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089719    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089785    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/452bd4ac-95e3-458c-830e-547c32d88ee0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089820    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/452bd4ac-95e3-458c-830e-547c32d88ee0-config-out\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089849    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.089909 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089869    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090188 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089936    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090188 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.089954    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090188 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.090026    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090188 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.090068    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-web-config\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090188 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.090099    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090188 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.090159    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090391 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.090198    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090391 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.090218    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdfvx\" (UniqueName: \"kubernetes.io/projected/452bd4ac-95e3-458c-830e-547c32d88ee0-kube-api-access-fdfvx\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.090391 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.090247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.190681 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.190643    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.190681 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.190683    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.190943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.190704    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.190943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.190726    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-config\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.190943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.190754    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.190943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.190779    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.190805    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/452bd4ac-95e3-458c-830e-547c32d88ee0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191292    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/452bd4ac-95e3-458c-830e-547c32d88ee0-config-out\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191322    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191349    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191411    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191437    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191462    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191487    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-web-config\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191539    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191587    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191591    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191589    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191672    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191704    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdfvx\" (UniqueName: \"kubernetes.io/projected/452bd4ac-95e3-458c-830e-547c32d88ee0-kube-api-access-fdfvx\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191744    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191785    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.192091 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.191863    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.193069 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.192006    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.193069 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.192458    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.194937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.193798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.194937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.194143    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/452bd4ac-95e3-458c-830e-547c32d88ee0-config-out\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.194937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.194285    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.194937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.194704    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.196476 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.196433    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-web-config\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.196992 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.196967    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.198138 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.197667    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/452bd4ac-95e3-458c-830e-547c32d88ee0-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.198138 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.197809    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.198138 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.197916    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-config\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.198336 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.198239    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/452bd4ac-95e3-458c-830e-547c32d88ee0-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.198336 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.198266    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.198424 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.198399    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.198570 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.198549    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.199358 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.199339    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/452bd4ac-95e3-458c-830e-547c32d88ee0-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.202768 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.202747    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdfvx\" (UniqueName: \"kubernetes.io/projected/452bd4ac-95e3-458c-830e-547c32d88ee0-kube-api-access-fdfvx\") pod \"prometheus-k8s-0\" (UID: \"452bd4ac-95e3-458c-830e-547c32d88ee0\") " pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.303864 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.303823    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:03.453276 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.453242    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Jul 14 00:49:03.456480 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:49:03.456448    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod452bd4ac_95e3_458c_830e_547c32d88ee0.slice/crio-06752cf303db6975bb3eeba25d9f3c3d95654435bf7c6c9221ea3bc14ee3a2a0 WatchSource:0}: Error finding container 06752cf303db6975bb3eeba25d9f3c3d95654435bf7c6c9221ea3bc14ee3a2a0: Status 404 returned error can't find the container with id 06752cf303db6975bb3eeba25d9f3c3d95654435bf7c6c9221ea3bc14ee3a2a0
Jul 14 00:49:03.560584 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.560543    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerStarted","Data":"7a6494193a25944a985a8de74dcce24cd558c5ec90dca4fdb0ac89bb7fd84a8f"}
Jul 14 00:49:03.560725 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.560664    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerStarted","Data":"b7dc1f34962905e5994fbf88bed6c3cd03c8c821be1c4ece09afbc3fd0f5f543"}
Jul 14 00:49:03.560725 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.560692    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerStarted","Data":"c1a45be8c465ac9c1776885b9ea768aacb570153b1a289a09aa838a9ca67b373"}
Jul 14 00:49:03.560725 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.560718    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerStarted","Data":"6493e1bcdc6a4fd722aa7f78d48369f01f6d99e6fc15271c5ae2f4b9430f6c2c"}
Jul 14 00:49:03.560893 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.560747    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerStarted","Data":"43809b9f9740d45b3b5c1b430481e506aea123f45d50cea0cd7ae732a433e0e5"}
Jul 14 00:49:03.562332 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.562306    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"f8ef372ad6f03bc19763f232a72d50ce7607cb18d91bf947063325ceff740f09"}
Jul 14 00:49:03.562450 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:03.562385    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"06752cf303db6975bb3eeba25d9f3c3d95654435bf7c6c9221ea3bc14ee3a2a0"}
Jul 14 00:49:04.520296 ip-10-0-129-125 kubenswrapper[2552]: E0714 00:49:04.520248    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-certs], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-multus/network-metrics-daemon-7bcw7" podUID="79b4894c-cac8-408b-ae28-e3c6788ba884"
Jul 14 00:49:04.568586 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:04.568547    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"f866da78-6c90-4a3a-83cd-3afed9cfd0a9","Type":"ContainerStarted","Data":"5a1c2d19bf3c5556b05eb61ea1abadfdefd5970f3f7d43555241647403c74eb7"}
Jul 14 00:49:04.569984 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:04.569958    2552 generic.go:356] "Generic (PLEG): container finished" podID="452bd4ac-95e3-458c-830e-547c32d88ee0" containerID="f8ef372ad6f03bc19763f232a72d50ce7607cb18d91bf947063325ceff740f09" exitCode=0
Jul 14 00:49:04.570128 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:04.570043    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerDied","Data":"f8ef372ad6f03bc19763f232a72d50ce7607cb18d91bf947063325ceff740f09"}
Jul 14 00:49:04.613940 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:04.613893    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.461867905 podStartE2EDuration="7.613877157s" totalImagesPullingTime="4.136971988s" totalInitContainerRuntime="15.037264ms" isStatefulPod=false podCreationTimestamp="2026-07-14 00:48:57 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:49:04.613738619 +0000 UTC m=+294.594823360" watchObservedRunningTime="2026-07-14 00:49:04.613877157 +0000 UTC m=+294.594961817"
Jul 14 00:49:07.537859 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:07.537823    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-qqm2z"
Jul 14 00:49:09.594082 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:09.594042    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"91e315ba6adc963d28ae0dc33543ef36610de31e50c306981572f7a4ad0d0aa5"}
Jul 14 00:49:09.594460 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:09.594127    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"1377ce41ad1420860e21ed0b5ed897dae943d2bfd4b79a59d9337ac7290ca3cd"}
Jul 14 00:49:10.549075 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:10.548904    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:49:10.557731 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:10.549278    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:49:10.557731 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:10.555699    2552 kubelet.go:1648] "Image garbage collection succeeded"
Jul 14 00:49:11.604573 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:11.604539    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"ce2aeaf917ea975e8d7a9fe407fc9d6b15576c346669500e8a4b3e464f246564"}
Jul 14 00:49:11.605044 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:11.604620    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"7dca78b80f360e0d87e79fc2b4867df49e7b4b84556d1a4361389a79fadc572c"}
Jul 14 00:49:11.605044 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:11.604661    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"c91f0b920424ec5409991b27bb8508a32f548cc3e4414acfe0bf6caa091eeb4c"}
Jul 14 00:49:11.605044 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:11.604704    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"452bd4ac-95e3-458c-830e-547c32d88ee0","Type":"ContainerStarted","Data":"f0da5aac76bd975622064352b28ff8730ebde2b24b943657b1465a84cece3d2d"}
Jul 14 00:49:11.634102 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:11.634018    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.710855962 podStartE2EDuration="9.633995602s" totalImagesPullingTime="5.892955592s" totalInitContainerRuntime="30.184048ms" isStatefulPod=false podCreationTimestamp="2026-07-14 00:49:02 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:49:11.630802428 +0000 UTC m=+301.611887089" watchObservedRunningTime="2026-07-14 00:49:11.633995602 +0000 UTC m=+301.615080267"
Jul 14 00:49:13.304519 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:13.304470    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:49:17.519222 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:17.519171    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:49:54.554867 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:49:54.554826    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" podUID="c26f12f8-a77a-4156-b3e4-6fe8c6cccb34" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 14 00:50:03.304546 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:03.304431    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:50:03.325050 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:03.325020    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:50:03.810912 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:03.810887    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Jul 14 00:50:04.554557 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:04.554521    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" podUID="c26f12f8-a77a-4156-b3e4-6fe8c6cccb34" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 14 00:50:14.555303 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:14.555261    2552 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" podUID="c26f12f8-a77a-4156-b3e4-6fe8c6cccb34" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Jul 14 00:50:14.555845 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:14.555341    2552 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt"
Jul 14 00:50:14.555845 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:14.555819    2552 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"a2b7eefa3157786511bb084390e73fce7aaf9ad85ba492c19e0667a67675deff"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Jul 14 00:50:14.555945 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:14.555868    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" podUID="c26f12f8-a77a-4156-b3e4-6fe8c6cccb34" containerName="service-proxy" containerID="cri-o://a2b7eefa3157786511bb084390e73fce7aaf9ad85ba492c19e0667a67675deff" gracePeriod=30
Jul 14 00:50:14.836660 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:14.836627    2552 generic.go:356] "Generic (PLEG): container finished" podID="c26f12f8-a77a-4156-b3e4-6fe8c6cccb34" containerID="a2b7eefa3157786511bb084390e73fce7aaf9ad85ba492c19e0667a67675deff" exitCode=2
Jul 14 00:50:14.836848 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:14.836694    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" event={"ID":"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34","Type":"ContainerDied","Data":"a2b7eefa3157786511bb084390e73fce7aaf9ad85ba492c19e0667a67675deff"}
Jul 14 00:50:14.836848 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:14.836745    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-7498d897dc-rvfzt" event={"ID":"c26f12f8-a77a-4156-b3e4-6fe8c6cccb34","Type":"ContainerStarted","Data":"c2bbad11555925c9f9595cf33a405c49f1702d220b831fd03999f04182865950"}
Jul 14 00:50:24.375066 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:24.375013    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:50:24.377239 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:24.377217    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/79b4894c-cac8-408b-ae28-e3c6788ba884-metrics-certs\") pod \"network-metrics-daemon-7bcw7\" (UID: \"79b4894c-cac8-408b-ae28-e3c6788ba884\") " pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:50:24.423063 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:24.423036    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-cmcmt\""
Jul 14 00:50:24.430626 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:24.430609    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-7bcw7"
Jul 14 00:50:24.550354 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:24.550254    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-7bcw7"]
Jul 14 00:50:24.552890 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:50:24.552862    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79b4894c_cac8_408b_ae28_e3c6788ba884.slice/crio-d821137b58dc30eaaae5617a5ebbd725c767bf370e33622fdc41292e06f1b0fe WatchSource:0}: Error finding container d821137b58dc30eaaae5617a5ebbd725c767bf370e33622fdc41292e06f1b0fe: Status 404 returned error can't find the container with id d821137b58dc30eaaae5617a5ebbd725c767bf370e33622fdc41292e06f1b0fe
Jul 14 00:50:24.554935 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:24.554919    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:50:24.871678 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:24.871646    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bcw7" event={"ID":"79b4894c-cac8-408b-ae28-e3c6788ba884","Type":"ContainerStarted","Data":"d821137b58dc30eaaae5617a5ebbd725c767bf370e33622fdc41292e06f1b0fe"}
Jul 14 00:50:25.876705 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:25.876648    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bcw7" event={"ID":"79b4894c-cac8-408b-ae28-e3c6788ba884","Type":"ContainerStarted","Data":"ee429ea46536f3282903e1f0a64950f6f706690a90bb4de65f0b2d403a38f3ca"}
Jul 14 00:50:25.877050 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:25.876725    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-7bcw7" event={"ID":"79b4894c-cac8-408b-ae28-e3c6788ba884","Type":"ContainerStarted","Data":"b18ea62511d7be4307bc6e004c74314ff4730c8eec8cfede362a53f292519425"}
Jul 14 00:50:25.891175 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:50:25.891121    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-7bcw7" podStartSLOduration=374.026416165 podStartE2EDuration="6m14.891104676s" totalImagesPullingTime="864.688511ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:44:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:50:25.890953693 +0000 UTC m=+375.872038354" watchObservedRunningTime="2026-07-14 00:50:25.891104676 +0000 UTC m=+375.872189340"
Jul 14 00:52:40.759472 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.759432    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"]
Jul 14 00:52:40.762838 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.762812    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.768267 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.767936    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Jul 14 00:52:40.768267 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.767965    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Jul 14 00:52:40.768267 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.767942    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-f5mkw\""
Jul 14 00:52:40.768267 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.767946    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Jul 14 00:52:40.768267 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.768110    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Jul 14 00:52:40.768267 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.768217    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Jul 14 00:52:40.776168 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.775377    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"]
Jul 14 00:52:40.850975 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.850943    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-cert\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.851134 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.850982    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5cllm\" (UniqueName: \"kubernetes.io/projected/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-kube-api-access-5cllm\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.851134 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.851050    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-metrics-cert\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.851134 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.851102    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-manager-config\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.952461 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.952422    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-cert\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.952461 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.952465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5cllm\" (UniqueName: \"kubernetes.io/projected/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-kube-api-access-5cllm\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.952731 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.952486    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-metrics-cert\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.952731 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.952541    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-manager-config\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.953121 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.953102    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-manager-config\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.954952 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.954931    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-metrics-cert\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.955039 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.954969    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-cert\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:40.968293 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:40.968266    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5cllm\" (UniqueName: \"kubernetes.io/projected/8b92f0f2-2bd3-42b4-862b-bc17acaa08e1-kube-api-access-5cllm\") pod \"lws-controller-manager-5bd544bbf-pbkhm\" (UID: \"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1\") " pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:41.078044 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.078014    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:41.210166 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.210125    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"]
Jul 14 00:52:41.213077 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:52:41.213052    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b92f0f2_2bd3_42b4_862b_bc17acaa08e1.slice/crio-219b672a7137070a7c22e47aa2214a521d95936633b89b93c26607bf0825ad6a WatchSource:0}: Error finding container 219b672a7137070a7c22e47aa2214a521d95936633b89b93c26607bf0825ad6a: Status 404 returned error can't find the container with id 219b672a7137070a7c22e47aa2214a521d95936633b89b93c26607bf0825ad6a
Jul 14 00:52:41.361081 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.360997    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm" event={"ID":"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1","Type":"ContainerStarted","Data":"219b672a7137070a7c22e47aa2214a521d95936633b89b93c26607bf0825ad6a"}
Jul 14 00:52:41.856326 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.856283    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"]
Jul 14 00:52:41.860580 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.860558    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:41.865322 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.865304    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 14 00:52:41.865424 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.865390    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 14 00:52:41.865727 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.865709    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 14 00:52:41.867943 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.867929    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 14 00:52:41.873354 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.873338    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-k2lvq\""
Jul 14 00:52:41.879123 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.879105    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"]
Jul 14 00:52:41.960714 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.960687    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-tmp\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:41.960853 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.960730    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:41.960853 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.960785    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzsxx\" (UniqueName: \"kubernetes.io/projected/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-kube-api-access-hzsxx\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:41.960853 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.960848    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-manifests\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:41.960949 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:41.960867    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.062105 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.062073    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.062105 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.062121    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzsxx\" (UniqueName: \"kubernetes.io/projected/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-kube-api-access-hzsxx\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.062363 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.062190    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-manifests\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.062363 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.062211    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.062363 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.062243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-tmp\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.062719 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.062691    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-manifests\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.065000 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.064976    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-apiservice-cert\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.065122 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.065066    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-tmp\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.065184 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.065130    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-webhook-cert\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.072822 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.072799    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzsxx\" (UniqueName: \"kubernetes.io/projected/1eb3d285-aa44-4bf9-b85b-81ed74cb1be1-kube-api-access-hzsxx\") pod \"opendatahub-operator-controller-manager-6b7d6985f7-s4wp8\" (UID: \"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1\") " pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.170900 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.170810    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:42.328723 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.328637    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"]
Jul 14 00:52:42.330849 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:52:42.330825    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1eb3d285_aa44_4bf9_b85b_81ed74cb1be1.slice/crio-f213f77c16231a07a831927a421db0a781b5522534f689b553db78ea5e6b4ab2 WatchSource:0}: Error finding container f213f77c16231a07a831927a421db0a781b5522534f689b553db78ea5e6b4ab2: Status 404 returned error can't find the container with id f213f77c16231a07a831927a421db0a781b5522534f689b553db78ea5e6b4ab2
Jul 14 00:52:42.369049 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:42.369009    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8" event={"ID":"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1","Type":"ContainerStarted","Data":"f213f77c16231a07a831927a421db0a781b5522534f689b553db78ea5e6b4ab2"}
Jul 14 00:52:46.390232 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:46.390185    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm" event={"ID":"8b92f0f2-2bd3-42b4-862b-bc17acaa08e1","Type":"ContainerStarted","Data":"fd0424b0a70f3b0190e5e6f9677da871e5b2f01d36cba74429aa53bade6b5ac3"}
Jul 14 00:52:46.390754 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:46.390365    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:46.391856 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:46.391832    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8" event={"ID":"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1","Type":"ContainerStarted","Data":"82f6c42a17405dfc2c0b5427c907e0f19e0d6882dbb415db594125091c64b6f1"}
Jul 14 00:52:46.431662 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:46.431605    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm" podStartSLOduration=1.814401851 podStartE2EDuration="6.431591126s" totalImagesPullingTime="4.617189275s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:52:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:52:46.429975327 +0000 UTC m=+516.411059991" watchObservedRunningTime="2026-07-14 00:52:46.431591126 +0000 UTC m=+516.412675787"
Jul 14 00:52:47.397613 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:47.397574    2552 generic.go:356] "Generic (PLEG): container finished" podID="1eb3d285-aa44-4bf9-b85b-81ed74cb1be1" containerID="82f6c42a17405dfc2c0b5427c907e0f19e0d6882dbb415db594125091c64b6f1" exitCode=0
Jul 14 00:52:47.398095 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:47.397665    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8" event={"ID":"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1","Type":"ContainerDied","Data":"82f6c42a17405dfc2c0b5427c907e0f19e0d6882dbb415db594125091c64b6f1"}
Jul 14 00:52:48.403115 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:48.403084    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8" event={"ID":"1eb3d285-aa44-4bf9-b85b-81ed74cb1be1","Type":"ContainerStarted","Data":"ca07583bebec5104666e4e5bfec96ff4203a7cf448725a6401707263a2ee1213"}
Jul 14 00:52:48.403521 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:48.403205    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:52:48.426282 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:48.426226    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8" podStartSLOduration=2.405984668 podStartE2EDuration="7.426203491s" totalImagesPullingTime="3.935493705s" totalInitContainerRuntime="1.084725118s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:52:41 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:52:48.423921517 +0000 UTC m=+518.405006178" watchObservedRunningTime="2026-07-14 00:52:48.426203491 +0000 UTC m=+518.407288153"
Jul 14 00:52:57.399882 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:57.399850    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bd544bbf-pbkhm"
Jul 14 00:52:59.409463 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:52:59.409431    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-6b7d6985f7-s4wp8"
Jul 14 00:53:02.688533 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.688438    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"]
Jul 14 00:53:02.702937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.702911    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.705611 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.705584    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 14 00:53:02.705757 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.705642    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 14 00:53:02.706763 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.706742    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 14 00:53:02.706858 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.706783    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 14 00:53:02.706858 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.706800    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-spnm7\""
Jul 14 00:53:02.712072 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.712045    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"]
Jul 14 00:53:02.846345 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.846311    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9tz7\" (UniqueName: \"kubernetes.io/projected/93bb0628-6a62-4442-a27e-bd8e614434b5-kube-api-access-z9tz7\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.846544 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.846373    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0628-6a62-4442-a27e-bd8e614434b5-tls-certs\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.846544 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.846492    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bb0628-6a62-4442-a27e-bd8e614434b5-tmp\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.947901 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.947806    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bb0628-6a62-4442-a27e-bd8e614434b5-tmp\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.947901 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.947862    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9tz7\" (UniqueName: \"kubernetes.io/projected/93bb0628-6a62-4442-a27e-bd8e614434b5-kube-api-access-z9tz7\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.947901 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.947905    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0628-6a62-4442-a27e-bd8e614434b5-tls-certs\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.950233 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.950203    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/93bb0628-6a62-4442-a27e-bd8e614434b5-tmp\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.950536 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.950488    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/93bb0628-6a62-4442-a27e-bd8e614434b5-tls-certs\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:02.962705 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:02.962673    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9tz7\" (UniqueName: \"kubernetes.io/projected/93bb0628-6a62-4442-a27e-bd8e614434b5-kube-api-access-z9tz7\") pod \"kube-auth-proxy-5f876ccc94-wrs6n\" (UID: \"93bb0628-6a62-4442-a27e-bd8e614434b5\") " pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:03.014292 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:03.014259    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"
Jul 14 00:53:03.146297 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:03.146271    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n"]
Jul 14 00:53:03.150278 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:53:03.149383    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93bb0628_6a62_4442_a27e_bd8e614434b5.slice/crio-888e7f9f6bd6c30101551d33077945488dc2c8464d8763b5582e5910a92bb218 WatchSource:0}: Error finding container 888e7f9f6bd6c30101551d33077945488dc2c8464d8763b5582e5910a92bb218: Status 404 returned error can't find the container with id 888e7f9f6bd6c30101551d33077945488dc2c8464d8763b5582e5910a92bb218
Jul 14 00:53:03.462286 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:03.462246    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n" event={"ID":"93bb0628-6a62-4442-a27e-bd8e614434b5","Type":"ContainerStarted","Data":"888e7f9f6bd6c30101551d33077945488dc2c8464d8763b5582e5910a92bb218"}
Jul 14 00:53:06.476525 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:06.476475    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n" event={"ID":"93bb0628-6a62-4442-a27e-bd8e614434b5","Type":"ContainerStarted","Data":"f96a007d924366ed3de9e20c7147eed88fd260090d638aa064c78b885d2bb308"}
Jul 14 00:53:06.493863 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:53:06.493818    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-5f876ccc94-wrs6n" podStartSLOduration=1.674393786 podStartE2EDuration="4.493804453s" totalImagesPullingTime="2.819410667s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:53:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:53:06.492544518 +0000 UTC m=+536.473629180" watchObservedRunningTime="2026-07-14 00:53:06.493804453 +0000 UTC m=+536.474889152"
Jul 14 00:54:10.574558 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:10.574530    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:54:10.576847 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:10.575407    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:54:47.436253 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.436166    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"]
Jul 14 00:54:47.439340 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.439319    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"
Jul 14 00:54:47.441908 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.441884    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-gdz9q\""
Jul 14 00:54:47.442414 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.442395    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 14 00:54:47.442897 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.442882    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 14 00:54:47.442979 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.442962    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 14 00:54:47.450252 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.450231    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"]
Jul 14 00:54:47.591192 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.591160    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq989\" (UniqueName: \"kubernetes.io/projected/a7312b34-36e0-49cc-83b2-f4f5ece254e0-kube-api-access-qq989\") pod \"dns-operator-controller-manager-864dd78c4c-9wv7m\" (UID: \"a7312b34-36e0-49cc-83b2-f4f5ece254e0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"
Jul 14 00:54:47.692092 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.692017    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq989\" (UniqueName: \"kubernetes.io/projected/a7312b34-36e0-49cc-83b2-f4f5ece254e0-kube-api-access-qq989\") pod \"dns-operator-controller-manager-864dd78c4c-9wv7m\" (UID: \"a7312b34-36e0-49cc-83b2-f4f5ece254e0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"
Jul 14 00:54:47.700270 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.700235    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq989\" (UniqueName: \"kubernetes.io/projected/a7312b34-36e0-49cc-83b2-f4f5ece254e0-kube-api-access-qq989\") pod \"dns-operator-controller-manager-864dd78c4c-9wv7m\" (UID: \"a7312b34-36e0-49cc-83b2-f4f5ece254e0\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"
Jul 14 00:54:47.749650 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.749611    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"
Jul 14 00:54:47.870907 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:47.870879    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"]
Jul 14 00:54:47.873790 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:54:47.873765    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7312b34_36e0_49cc_83b2_f4f5ece254e0.slice/crio-bf3e3ebeb12fb7aff2a296c694994c569947b181da59c926060d1f1e33d21979 WatchSource:0}: Error finding container bf3e3ebeb12fb7aff2a296c694994c569947b181da59c926060d1f1e33d21979: Status 404 returned error can't find the container with id bf3e3ebeb12fb7aff2a296c694994c569947b181da59c926060d1f1e33d21979
Jul 14 00:54:48.870709 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:48.870675    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m" event={"ID":"a7312b34-36e0-49cc-83b2-f4f5ece254e0","Type":"ContainerStarted","Data":"bf3e3ebeb12fb7aff2a296c694994c569947b181da59c926060d1f1e33d21979"}
Jul 14 00:54:50.882009 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:50.881972    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m" event={"ID":"a7312b34-36e0-49cc-83b2-f4f5ece254e0","Type":"ContainerStarted","Data":"e0c73adaae75ed83178de5f3f87b6c9336949b4680d73de31654369f777a039a"}
Jul 14 00:54:50.882368 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:50.882102    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"
Jul 14 00:54:50.900213 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:50.900166    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m" podStartSLOduration=1.521872243 podStartE2EDuration="3.900152679s" totalImagesPullingTime="2.378280436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:54:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:54:50.89929427 +0000 UTC m=+640.880378932" watchObservedRunningTime="2026-07-14 00:54:50.900152679 +0000 UTC m=+640.881237380"
Jul 14 00:54:54.349409 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.349373    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"]
Jul 14 00:54:54.352837 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.352819    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.355609 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.355583    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Jul 14 00:54:54.355715 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.355589    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Jul 14 00:54:54.355715 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.355661    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-n2brf\""
Jul 14 00:54:54.362949 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.362928    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"]
Jul 14 00:54:54.444834 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.444797    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqrft\" (UniqueName: \"kubernetes.io/projected/adb80f30-3030-45b7-80b0-f74b99667170-kube-api-access-pqrft\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.444995 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.444837    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/adb80f30-3030-45b7-80b0-f74b99667170-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.444995 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.444871    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adb80f30-3030-45b7-80b0-f74b99667170-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.545306 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.545272    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqrft\" (UniqueName: \"kubernetes.io/projected/adb80f30-3030-45b7-80b0-f74b99667170-kube-api-access-pqrft\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.545306 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.545311    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/adb80f30-3030-45b7-80b0-f74b99667170-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.545530 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.545333    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adb80f30-3030-45b7-80b0-f74b99667170-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.545989 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.545968    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/adb80f30-3030-45b7-80b0-f74b99667170-nginx-conf\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.547797 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.547778    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/adb80f30-3030-45b7-80b0-f74b99667170-plugin-serving-cert\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.553846 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.553821    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqrft\" (UniqueName: \"kubernetes.io/projected/adb80f30-3030-45b7-80b0-f74b99667170-kube-api-access-pqrft\") pod \"kuadrant-console-plugin-66b5774985-nzhvx\" (UID: \"adb80f30-3030-45b7-80b0-f74b99667170\") " pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.661159 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.661080    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"
Jul 14 00:54:54.799192 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.799167    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx"]
Jul 14 00:54:54.801678 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:54:54.801649    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podadb80f30_3030_45b7_80b0_f74b99667170.slice/crio-5e9f02aac587cebf0b6a6a9aad7829ec7718ccf3e5af8809bb22b3748058ea35 WatchSource:0}: Error finding container 5e9f02aac587cebf0b6a6a9aad7829ec7718ccf3e5af8809bb22b3748058ea35: Status 404 returned error can't find the container with id 5e9f02aac587cebf0b6a6a9aad7829ec7718ccf3e5af8809bb22b3748058ea35
Jul 14 00:54:54.897937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:54:54.897905    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx" event={"ID":"adb80f30-3030-45b7-80b0-f74b99667170","Type":"ContainerStarted","Data":"5e9f02aac587cebf0b6a6a9aad7829ec7718ccf3e5af8809bb22b3748058ea35"}
Jul 14 00:55:01.887937 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:01.887903    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-9wv7m"
Jul 14 00:55:19.014462 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:19.014423    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx" event={"ID":"adb80f30-3030-45b7-80b0-f74b99667170","Type":"ContainerStarted","Data":"fb466579df08d2ef435000faad3567bb35de2139c331f04e29b5b407b41ec9aa"}
Jul 14 00:55:19.031141 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:19.031096    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66b5774985-nzhvx" podStartSLOduration=1.6208453619999998 podStartE2EDuration="25.031083907s" totalImagesPullingTime="23.410238545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:54:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:55:19.029096408 +0000 UTC m=+669.010181070" watchObservedRunningTime="2026-07-14 00:55:19.031083907 +0000 UTC m=+669.012168623"
Jul 14 00:55:51.288529 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.288479    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:55:51.362486 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.362454    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:55:51.362652 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.362492    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:55:51.362652 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.362576    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.365681 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.365662    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 14 00:55:51.439085 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.439057    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7199c424-9674-4941-84e1-1e072b30ea67-config-file\") pod \"limitador-limitador-54d6d988d5-b9zjq\" (UID: \"7199c424-9674-4941-84e1-1e072b30ea67\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.439225 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.439118    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k5wpb\" (UniqueName: \"kubernetes.io/projected/7199c424-9674-4941-84e1-1e072b30ea67-kube-api-access-k5wpb\") pod \"limitador-limitador-54d6d988d5-b9zjq\" (UID: \"7199c424-9674-4941-84e1-1e072b30ea67\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.539993 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.539928    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7199c424-9674-4941-84e1-1e072b30ea67-config-file\") pod \"limitador-limitador-54d6d988d5-b9zjq\" (UID: \"7199c424-9674-4941-84e1-1e072b30ea67\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.540107 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.540010    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k5wpb\" (UniqueName: \"kubernetes.io/projected/7199c424-9674-4941-84e1-1e072b30ea67-kube-api-access-k5wpb\") pod \"limitador-limitador-54d6d988d5-b9zjq\" (UID: \"7199c424-9674-4941-84e1-1e072b30ea67\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.540579 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.540563    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/7199c424-9674-4941-84e1-1e072b30ea67-config-file\") pod \"limitador-limitador-54d6d988d5-b9zjq\" (UID: \"7199c424-9674-4941-84e1-1e072b30ea67\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.548265 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.548240    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k5wpb\" (UniqueName: \"kubernetes.io/projected/7199c424-9674-4941-84e1-1e072b30ea67-kube-api-access-k5wpb\") pod \"limitador-limitador-54d6d988d5-b9zjq\" (UID: \"7199c424-9674-4941-84e1-1e072b30ea67\") " pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.672418 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.672380    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:51.798133 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.798110    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:55:51.800891 ip-10-0-129-125 kubenswrapper[2552]: W0714 00:55:51.800859    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7199c424_9674_4941_84e1_1e072b30ea67.slice/crio-f6bb66439d5664779856921cfb9fbdb5007ee664069b504bd2e978ad3d79eee2 WatchSource:0}: Error finding container f6bb66439d5664779856921cfb9fbdb5007ee664069b504bd2e978ad3d79eee2: Status 404 returned error can't find the container with id f6bb66439d5664779856921cfb9fbdb5007ee664069b504bd2e978ad3d79eee2
Jul 14 00:55:51.802760 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:51.802744    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 00:55:52.152775 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:52.152739    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq" event={"ID":"7199c424-9674-4941-84e1-1e072b30ea67","Type":"ContainerStarted","Data":"f6bb66439d5664779856921cfb9fbdb5007ee664069b504bd2e978ad3d79eee2"}
Jul 14 00:55:55.168046 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:55.168013    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq" event={"ID":"7199c424-9674-4941-84e1-1e072b30ea67","Type":"ContainerStarted","Data":"0862b8ed3179b5c6a19e2cc650318112ae1089e1470d81da68857b30c90b7f00"}
Jul 14 00:55:55.168408 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:55.168140    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:55:55.184311 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:55:55.184264    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq" podStartSLOduration=1.72494247 podStartE2EDuration="4.184248249s" totalImagesPullingTime="2.459305779s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:55:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:55:55.183094132 +0000 UTC m=+705.164178793" watchObservedRunningTime="2026-07-14 00:55:55.184248249 +0000 UTC m=+705.165332910"
Jul 14 00:56:06.174254 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:56:06.174179    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"
Jul 14 00:57:08.415283 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:57:08.415245    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:57:26.060110 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:57:26.060070    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:57:37.428689 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:57:37.428611    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:57:57.821870 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:57:57.821837    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:58:14.730481 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:14.730448    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:58:20.020733 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.020702    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"]
Jul 14 00:58:20.023984 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.023969    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.026712 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.026681    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 14 00:58:20.027992 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.027969    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-k75jk\""
Jul 14 00:58:20.028095 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.028020    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 14 00:58:20.028095 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.027973    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 14 00:58:20.037993 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.037970    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"]
Jul 14 00:58:20.092358 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.092330    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.092488 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.092371    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ts5h4\" (UniqueName: \"kubernetes.io/projected/7bdab809-df29-4923-a9f3-bb4487b83069-kube-api-access-ts5h4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.092488 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.092397    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.092488 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.092449    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdab809-df29-4923-a9f3-bb4487b83069-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.092488 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.092486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.092665 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.092567    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.193639 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.193602    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.193786 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.193667    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.193786 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.193690    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ts5h4\" (UniqueName: \"kubernetes.io/projected/7bdab809-df29-4923-a9f3-bb4487b83069-kube-api-access-ts5h4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.193786 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.193709    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.193786 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.193730    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdab809-df29-4923-a9f3-bb4487b83069-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.193786 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.193751    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.194068 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.194044    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.194123 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.194094    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.194173 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.194102    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.195941 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.195921    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7bdab809-df29-4923-a9f3-bb4487b83069-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.196208 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.196189    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7bdab809-df29-4923-a9f3-bb4487b83069-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.210674 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.210648    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ts5h4\" (UniqueName: \"kubernetes.io/projected/7bdab809-df29-4923-a9f3-bb4487b83069-kube-api-access-ts5h4\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-dxfrg\" (UID: \"7bdab809-df29-4923-a9f3-bb4487b83069\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.335512 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.335468    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:20.463597 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.463429    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"]
Jul 14 00:58:20.792214 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:20.792140    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg" event={"ID":"7bdab809-df29-4923-a9f3-bb4487b83069","Type":"ContainerStarted","Data":"99de24e361ad7c511d7b744d5190950d8c79981deb3ff8c7b33ae850dbb7b0f6"}
Jul 14 00:58:21.798819 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:21.798728    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg" event={"ID":"7bdab809-df29-4923-a9f3-bb4487b83069","Type":"ContainerStarted","Data":"572a67074306a9a0e83b487e01908aacf87abcc201bce9a1e55c7d65d1929afa"}
Jul 14 00:58:21.817107 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:21.817051    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg" podStartSLOduration=0.857873396 podStartE2EDuration="1.817032284s" totalImagesPullingTime="959.158888ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 00:58:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 00:58:21.816584039 +0000 UTC m=+851.797668704" watchObservedRunningTime="2026-07-14 00:58:21.817032284 +0000 UTC m=+851.798116981"
Jul 14 00:58:22.026657 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:22.026625    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:58:30.335766 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:30.335731    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:30.336185 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:30.335803    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:30.348737 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:30.348713    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:30.846851 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:30.846823    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-dxfrg"
Jul 14 00:58:37.532884 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:58:37.532851    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 00:59:10.601817 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:59:10.601716    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 00:59:10.602924 ip-10-0-129-125 kubenswrapper[2552]: I0714 00:59:10.602781    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:00:12.642701 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:00:12.642666    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:00:22.729219 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:00:22.729182    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:00:34.026759 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:00:34.026688    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:00:42.730187 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:00:42.730154    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:00:53.833225 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:00:53.833194    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:01:21.347010 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:01:21.346977    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:01:34.827274 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:01:34.827244    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:02:08.831831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:02:08.831757    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:02:22.224388 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:02:22.224357    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:02:32.223733 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:02:32.223699    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:02:41.735218 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:02:41.735179    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:02:52.324858 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:02:52.324825    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:03:01.234203 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:03:01.234161    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:03:12.525069 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:03:12.525040    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:03:22.119080 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:03:22.119046    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:03:44.633335 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:03:44.633262    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:03:55.326068 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:03:55.326037    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:04:10.628304 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:04:10.628203    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:04:10.632345 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:04:10.631183    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:04:16.233154 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:04:16.233118    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:04:28.730759 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:04:28.730723    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:04:44.539096 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:04:44.539062    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:04:54.033742 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:04:54.033704    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:05:03.523893 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:05:03.523804    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:05:13.229807 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:05:13.229772    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:05:22.626099 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:05:22.626065    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:07:10.029304 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:07:10.029230    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:07:20.129902 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:07:20.129869    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:07:32.832876 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:07:32.832844    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:07:43.424774 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:07:43.424739    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:07:55.423218 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:07:55.423179    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:08:04.624337 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:08:04.624250    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:08:14.734293 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:08:14.734254    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:08:25.121057 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:08:25.121022    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:08:35.829408 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:08:35.829373    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:08:45.529401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:08:45.529368    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:08:58.140278 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:08:58.140242    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:09:10.655287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:09:10.655171    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:09:10.658492 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:09:10.658382    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:09:19.833804 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:09:19.833771    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:09:28.929616 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:09:28.929581    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:09:39.234530 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:09:39.234432    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:09:50.332757 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:09:50.332724    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:10:07.530950 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:10:07.530908    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:10:17.127748 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:10:17.127715    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:10:26.132627 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:10:26.132592    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:10:36.934105 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:10:36.934068    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:10:46.031407 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:10:46.031373    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:10:55.429893 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:10:55.429857    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:11:09.825647 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:11:09.825559    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:11:18.729446 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:11:18.729411    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:11:28.236259 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:11:28.236220    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:11:40.422153 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:11:40.422116    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:11:49.737333 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:11:49.737297    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:12:00.239155 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:12:00.239118    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:12:13.834030 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:12:13.833987    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:12:25.730740 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:12:25.730699    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:12:37.230538 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:12:37.230428    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:12:48.627047 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:12:48.627013    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:12:57.421277 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:12:57.421240    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:13:07.123680 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:07.123645    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:13:10.095844 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.095811    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"]
Jul 14 01:13:10.099337 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.099316    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.102025 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.101999    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-b9a4f161-serving-cert\""
Jul 14 01:13:10.102180 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.102123    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:13:10.103298 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.103279    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:13:10.103298 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.103291    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:13:10.108641 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.108616    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"]
Jul 14 01:13:10.202686 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.202647    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqx6x\" (UniqueName: \"kubernetes.io/projected/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-kube-api-access-zqx6x\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.202686 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.202683    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-openshift-service-ca\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.202888 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.202727    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.303790 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.303751    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqx6x\" (UniqueName: \"kubernetes.io/projected/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-kube-api-access-zqx6x\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.303790 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.303793    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-openshift-service-ca\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.304058 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.303838    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.304058 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:13:10.303964    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-serving-cert: secret "maas-api-e2e-ait-b9a4f161-serving-cert" not found
Jul 14 01:13:10.304058 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:13:10.304029    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls podName:d718d112-3c2d-4aa8-bd9e-dfc5399b0150 nodeName:}" failed. No retries permitted until 2026-07-14 01:13:10.804011715 +0000 UTC m=+1740.785096355 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls") pod "maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp" (UID: "d718d112-3c2d-4aa8-bd9e-dfc5399b0150") : secret "maas-api-e2e-ait-b9a4f161-serving-cert" not found
Jul 14 01:13:10.304464 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.304443    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-openshift-service-ca\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.315149 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.315123    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqx6x\" (UniqueName: \"kubernetes.io/projected/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-kube-api-access-zqx6x\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.808376 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.808342    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:10.810485 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:10.810467    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls\") pod \"maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:11.013080 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:11.013048    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:13:11.021478 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:11.021447    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:11.152737 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:11.152648    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"]
Jul 14 01:13:11.155626 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:13:11.155599    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd718d112_3c2d_4aa8_bd9e_dfc5399b0150.slice/crio-f9406be2bff812469ad2ae84e473feed9c3c3c50b83b07f8a6e46cb6cd0b29a8 WatchSource:0}: Error finding container f9406be2bff812469ad2ae84e473feed9c3c3c50b83b07f8a6e46cb6cd0b29a8: Status 404 returned error can't find the container with id f9406be2bff812469ad2ae84e473feed9c3c3c50b83b07f8a6e46cb6cd0b29a8
Jul 14 01:13:11.157287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:11.157268    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:13:11.665570 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:11.665534    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp" event={"ID":"d718d112-3c2d-4aa8-bd9e-dfc5399b0150","Type":"ContainerStarted","Data":"f9406be2bff812469ad2ae84e473feed9c3c3c50b83b07f8a6e46cb6cd0b29a8"}
Jul 14 01:13:13.677638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:13.677535    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp" event={"ID":"d718d112-3c2d-4aa8-bd9e-dfc5399b0150","Type":"ContainerStarted","Data":"a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6"}
Jul 14 01:13:13.677638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:13.677633    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:13.693747 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:13.693695    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp" podStartSLOduration=1.4352660259999999 podStartE2EDuration="3.693680797s" totalImagesPullingTime="2.258414771s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:13:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:13:13.692375971 +0000 UTC m=+1743.673460624" watchObservedRunningTime="2026-07-14 01:13:13.693680797 +0000 UTC m=+1743.674765504"
Jul 14 01:13:14.891489 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:14.891457    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"]
Jul 14 01:13:14.894806 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:14.894786    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"
Jul 14 01:13:14.897482 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:14.897458    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:13:14.902433 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:14.902411    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"]
Jul 14 01:13:15.043001 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:15.042967    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babbf821-5b2e-4367-8df6-88d055beeaa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2\" (UID: \"babbf821-5b2e-4367-8df6-88d055beeaa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"
Jul 14 01:13:15.143817 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:15.143742    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babbf821-5b2e-4367-8df6-88d055beeaa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2\" (UID: \"babbf821-5b2e-4367-8df6-88d055beeaa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"
Jul 14 01:13:15.144390 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:15.144369    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babbf821-5b2e-4367-8df6-88d055beeaa7-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2\" (UID: \"babbf821-5b2e-4367-8df6-88d055beeaa7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"
Jul 14 01:13:15.206236 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:15.206190    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"
Jul 14 01:13:15.333854 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:15.333836    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"]
Jul 14 01:13:15.336037 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:13:15.336017    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabbf821_5b2e_4367_8df6_88d055beeaa7.slice/crio-cfb3d567a5e3a38813838b2d2aee064b6e4adea38e6da0fb9c823912847e30de WatchSource:0}: Error finding container cfb3d567a5e3a38813838b2d2aee064b6e4adea38e6da0fb9c823912847e30de: Status 404 returned error can't find the container with id cfb3d567a5e3a38813838b2d2aee064b6e4adea38e6da0fb9c823912847e30de
Jul 14 01:13:15.442222 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:15.442142    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:13:15.687353 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:15.687310    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" event={"ID":"babbf821-5b2e-4367-8df6-88d055beeaa7","Type":"ContainerStarted","Data":"cfb3d567a5e3a38813838b2d2aee064b6e4adea38e6da0fb9c823912847e30de"}
Jul 14 01:13:18.703102 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:18.703062    2552 generic.go:356] "Generic (PLEG): container finished" podID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerID="5f9c6e4e019e816d7d0522eff58b3203c426d2df4a1ebef4eb33f5417f3616d9" exitCode=7
Jul 14 01:13:18.703483 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:18.703146    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" event={"ID":"babbf821-5b2e-4367-8df6-88d055beeaa7","Type":"ContainerDied","Data":"5f9c6e4e019e816d7d0522eff58b3203c426d2df4a1ebef4eb33f5417f3616d9"}
Jul 14 01:13:18.703483 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:18.703338    2552 scope.go:117] "RemoveContainer" containerID="5f9c6e4e019e816d7d0522eff58b3203c426d2df4a1ebef4eb33f5417f3616d9"
Jul 14 01:13:19.686994 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:19.686926    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:13:19.709893 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:19.709854    2552 generic.go:356] "Generic (PLEG): container finished" podID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerID="4ea45c2800b404699efe03016bd61a72d46f5d819b225433877eea6f62e63eac" exitCode=7
Jul 14 01:13:19.710333 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:19.709932    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" event={"ID":"babbf821-5b2e-4367-8df6-88d055beeaa7","Type":"ContainerDied","Data":"4ea45c2800b404699efe03016bd61a72d46f5d819b225433877eea6f62e63eac"}
Jul 14 01:13:19.710333 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:19.709991    2552 scope.go:117] "RemoveContainer" containerID="5f9c6e4e019e816d7d0522eff58b3203c426d2df4a1ebef4eb33f5417f3616d9"
Jul 14 01:13:19.710333 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:19.710221    2552 scope.go:117] "RemoveContainer" containerID="4ea45c2800b404699efe03016bd61a72d46f5d819b225433877eea6f62e63eac"
Jul 14 01:13:19.710484 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:13:19.710461    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2_odh-ai-gateway-infra(babbf821-5b2e-4367-8df6-88d055beeaa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7"
Jul 14 01:13:20.716930 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:20.716903    2552 scope.go:117] "RemoveContainer" containerID="4ea45c2800b404699efe03016bd61a72d46f5d819b225433877eea6f62e63eac"
Jul 14 01:13:20.717294 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:13:20.717077    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2_odh-ai-gateway-infra(babbf821-5b2e-4367-8df6-88d055beeaa7)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7"
Jul 14 01:13:32.518963 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:32.518928    2552 scope.go:117] "RemoveContainer" containerID="4ea45c2800b404699efe03016bd61a72d46f5d819b225433877eea6f62e63eac"
Jul 14 01:13:32.772762 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:32.772695    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" event={"ID":"babbf821-5b2e-4367-8df6-88d055beeaa7","Type":"ContainerStarted","Data":"4ffebfa0a4dbfbc105d5b6fc8deb2ac2aba443596bac03cc9a0d136ae0bd322a"}
Jul 14 01:13:32.788287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:32.788236    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" podStartSLOduration=16.332994903 podStartE2EDuration="18.788222149s" totalImagesPullingTime="2.455227246s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:13:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:13:32.78672595 +0000 UTC m=+1762.767810623" watchObservedRunningTime="2026-07-14 01:13:32.788222149 +0000 UTC m=+1762.769306811"
Jul 14 01:13:33.779380 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:33.779346    2552 generic.go:356] "Generic (PLEG): container finished" podID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerID="4ffebfa0a4dbfbc105d5b6fc8deb2ac2aba443596bac03cc9a0d136ae0bd322a" exitCode=0
Jul 14 01:13:33.779380 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:33.779386    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" event={"ID":"babbf821-5b2e-4367-8df6-88d055beeaa7","Type":"ContainerDied","Data":"4ffebfa0a4dbfbc105d5b6fc8deb2ac2aba443596bac03cc9a0d136ae0bd322a"}
Jul 14 01:13:33.779851 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:33.779435    2552 scope.go:117] "RemoveContainer" containerID="4ea45c2800b404699efe03016bd61a72d46f5d819b225433877eea6f62e63eac"
Jul 14 01:13:33.796946 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:33.796921    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"]
Jul 14 01:13:34.909296 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:34.909274    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"
Jul 14 01:13:35.012389 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.012360    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babbf821-5b2e-4367-8df6-88d055beeaa7-maas-api-service-ca\") pod \"babbf821-5b2e-4367-8df6-88d055beeaa7\" (UID: \"babbf821-5b2e-4367-8df6-88d055beeaa7\") "
Jul 14 01:13:35.012755 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.012734    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babbf821-5b2e-4367-8df6-88d055beeaa7-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "babbf821-5b2e-4367-8df6-88d055beeaa7" (UID: "babbf821-5b2e-4367-8df6-88d055beeaa7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:13:35.113584 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.113556    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/babbf821-5b2e-4367-8df6-88d055beeaa7-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:13:35.791490 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.791463    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"
Jul 14 01:13:35.791689 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.791488    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2" event={"ID":"babbf821-5b2e-4367-8df6-88d055beeaa7","Type":"ContainerDied","Data":"cfb3d567a5e3a38813838b2d2aee064b6e4adea38e6da0fb9c823912847e30de"}
Jul 14 01:13:35.791689 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.791554    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfb3d567a5e3a38813838b2d2aee064b6e4adea38e6da0fb9c823912847e30de"
Jul 14 01:13:35.852397 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.852366    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"]
Jul 14 01:13:35.855912 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:35.855884    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-c98r2"]
Jul 14 01:13:36.524595 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:13:36.524565    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" path="/var/lib/kubelet/pods/babbf821-5b2e-4367-8df6-88d055beeaa7/volumes"
Jul 14 01:14:10.681811 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:10.681687    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:14:10.686119 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:10.685046    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:14:14.943230 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.943196    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"]
Jul 14 01:14:14.943638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.943539    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:14.943638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.943552    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:14.943638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.943567    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:14.943638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.943572    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:14.943638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.943632    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:14.943638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.943641    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:14.945457 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.945443    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"
Jul 14 01:14:14.948468 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.948445    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:14:14.959015 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.958991    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"]
Jul 14 01:14:14.968888 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:14.968866    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff072b7b-0a3e-48cc-bedc-823661c7ceb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9a4f161-knx78\" (UID: \"ff072b7b-0a3e-48cc-bedc-823661c7ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"
Jul 14 01:14:15.069703 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:15.069669    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff072b7b-0a3e-48cc-bedc-823661c7ceb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9a4f161-knx78\" (UID: \"ff072b7b-0a3e-48cc-bedc-823661c7ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"
Jul 14 01:14:15.070276 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:15.070259    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff072b7b-0a3e-48cc-bedc-823661c7ceb2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-b9a4f161-knx78\" (UID: \"ff072b7b-0a3e-48cc-bedc-823661c7ceb2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"
Jul 14 01:14:15.255610 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:15.255529    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"
Jul 14 01:14:15.379742 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:15.379715    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"]
Jul 14 01:14:15.381919 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:14:15.381888    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff072b7b_0a3e_48cc_bedc_823661c7ceb2.slice/crio-6e21db7ff46e7a1b5ff7ce760a5ebbf9220be5cb2140cd0876bc43c85b8d0164 WatchSource:0}: Error finding container 6e21db7ff46e7a1b5ff7ce760a5ebbf9220be5cb2140cd0876bc43c85b8d0164: Status 404 returned error can't find the container with id 6e21db7ff46e7a1b5ff7ce760a5ebbf9220be5cb2140cd0876bc43c85b8d0164
Jul 14 01:14:15.966999 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:15.966959    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78" event={"ID":"ff072b7b-0a3e-48cc-bedc-823661c7ceb2","Type":"ContainerStarted","Data":"c75d647dd43f098522e20820adcbb57a11d155b3ad669662c882b111bbc04698"}
Jul 14 01:14:15.967370 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:15.967022    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78" event={"ID":"ff072b7b-0a3e-48cc-bedc-823661c7ceb2","Type":"ContainerStarted","Data":"6e21db7ff46e7a1b5ff7ce760a5ebbf9220be5cb2140cd0876bc43c85b8d0164"}
Jul 14 01:14:15.984808 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:15.984758    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78" podStartSLOduration=1.9847433909999999 podStartE2EDuration="1.984743391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:14:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:14:15.982401055 +0000 UTC m=+1805.963485716" watchObservedRunningTime="2026-07-14 01:14:15.984743391 +0000 UTC m=+1805.965828051"
Jul 14 01:14:16.973202 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:16.973164    2552 generic.go:356] "Generic (PLEG): container finished" podID="ff072b7b-0a3e-48cc-bedc-823661c7ceb2" containerID="c75d647dd43f098522e20820adcbb57a11d155b3ad669662c882b111bbc04698" exitCode=0
Jul 14 01:14:16.973592 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:16.973248    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78" event={"ID":"ff072b7b-0a3e-48cc-bedc-823661c7ceb2","Type":"ContainerDied","Data":"c75d647dd43f098522e20820adcbb57a11d155b3ad669662c882b111bbc04698"}
Jul 14 01:14:18.098594 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:18.098571    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"
Jul 14 01:14:18.196212 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:18.196182    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff072b7b-0a3e-48cc-bedc-823661c7ceb2-maas-api-service-ca\") pod \"ff072b7b-0a3e-48cc-bedc-823661c7ceb2\" (UID: \"ff072b7b-0a3e-48cc-bedc-823661c7ceb2\") "
Jul 14 01:14:18.196547 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:18.196528    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ff072b7b-0a3e-48cc-bedc-823661c7ceb2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ff072b7b-0a3e-48cc-bedc-823661c7ceb2" (UID: "ff072b7b-0a3e-48cc-bedc-823661c7ceb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:14:18.296787 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:18.296703    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ff072b7b-0a3e-48cc-bedc-823661c7ceb2-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:14:18.982316 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:18.982238    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78"
Jul 14 01:14:18.982316 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:18.982258    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b9a4f161-knx78" event={"ID":"ff072b7b-0a3e-48cc-bedc-823661c7ceb2","Type":"ContainerDied","Data":"6e21db7ff46e7a1b5ff7ce760a5ebbf9220be5cb2140cd0876bc43c85b8d0164"}
Jul 14 01:14:18.982316 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:18.982301    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6e21db7ff46e7a1b5ff7ce760a5ebbf9220be5cb2140cd0876bc43c85b8d0164"
Jul 14 01:14:25.354432 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.354397    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"]
Jul 14 01:14:25.354911 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.354682    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp" podUID="d718d112-3c2d-4aa8-bd9e-dfc5399b0150" containerName="maas-api" containerID="cri-o://a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6" gracePeriod=30
Jul 14 01:14:25.594320 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.594300    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:14:25.660145 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.660057    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqx6x\" (UniqueName: \"kubernetes.io/projected/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-kube-api-access-zqx6x\") pod \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") "
Jul 14 01:14:25.660145 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.660144    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-openshift-service-ca\") pod \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") "
Jul 14 01:14:25.660363 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.660193    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls\") pod \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\" (UID: \"d718d112-3c2d-4aa8-bd9e-dfc5399b0150\") "
Jul 14 01:14:25.660570 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.660546    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d718d112-3c2d-4aa8-bd9e-dfc5399b0150" (UID: "d718d112-3c2d-4aa8-bd9e-dfc5399b0150"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:14:25.662232 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.662214    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-kube-api-access-zqx6x" (OuterVolumeSpecName: "kube-api-access-zqx6x") pod "d718d112-3c2d-4aa8-bd9e-dfc5399b0150" (UID: "d718d112-3c2d-4aa8-bd9e-dfc5399b0150"). InnerVolumeSpecName "kube-api-access-zqx6x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:14:25.662318 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.662298    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d718d112-3c2d-4aa8-bd9e-dfc5399b0150" (UID: "d718d112-3c2d-4aa8-bd9e-dfc5399b0150"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:14:25.760982 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.760945    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:14:25.760982 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.760974    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:14:25.760982 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:25.760987    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqx6x\" (UniqueName: \"kubernetes.io/projected/d718d112-3c2d-4aa8-bd9e-dfc5399b0150-kube-api-access-zqx6x\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:14:26.012764 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.012673    2552 generic.go:356] "Generic (PLEG): container finished" podID="d718d112-3c2d-4aa8-bd9e-dfc5399b0150" containerID="a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6" exitCode=0
Jul 14 01:14:26.012764 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.012714    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"
Jul 14 01:14:26.012978 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.012760    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp" event={"ID":"d718d112-3c2d-4aa8-bd9e-dfc5399b0150","Type":"ContainerDied","Data":"a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6"}
Jul 14 01:14:26.012978 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.012815    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp" event={"ID":"d718d112-3c2d-4aa8-bd9e-dfc5399b0150","Type":"ContainerDied","Data":"f9406be2bff812469ad2ae84e473feed9c3c3c50b83b07f8a6e46cb6cd0b29a8"}
Jul 14 01:14:26.012978 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.012836    2552 scope.go:117] "RemoveContainer" containerID="a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6"
Jul 14 01:14:26.027804 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.027785    2552 scope.go:117] "RemoveContainer" containerID="a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6"
Jul 14 01:14:26.028170 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:14:26.028142    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6\": container with ID starting with a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6 not found: ID does not exist" containerID="a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6"
Jul 14 01:14:26.028230 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.028178    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6"} err="failed to get container status \"a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6\": rpc error: code = NotFound desc = could not find container \"a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6\": container with ID starting with a5a396128c7f91d1029971b84a2cbb32b889e5615fcb9c03e6212dafab2510e6 not found: ID does not exist"
Jul 14 01:14:26.039881 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.039850    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"]
Jul 14 01:14:26.048144 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.048123    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-b9a4f161-577cb7784b-ppbrp"]
Jul 14 01:14:26.524181 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:26.524139    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d718d112-3c2d-4aa8-bd9e-dfc5399b0150" path="/var/lib/kubelet/pods/d718d112-3c2d-4aa8-bd9e-dfc5399b0150/volumes"
Jul 14 01:14:28.228817 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:28.228783    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:14:42.098470 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098438    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"]
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098770    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d718d112-3c2d-4aa8-bd9e-dfc5399b0150" containerName="maas-api"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098780    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d718d112-3c2d-4aa8-bd9e-dfc5399b0150" containerName="maas-api"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098795    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098800    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098807    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ff072b7b-0a3e-48cc-bedc-823661c7ceb2" containerName="revoke-keys"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098811    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ff072b7b-0a3e-48cc-bedc-823661c7ceb2" containerName="revoke-keys"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098859    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ff072b7b-0a3e-48cc-bedc-823661c7ceb2" containerName="revoke-keys"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098867    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d718d112-3c2d-4aa8-bd9e-dfc5399b0150" containerName="maas-api"
Jul 14 01:14:42.098891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.098875    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="babbf821-5b2e-4367-8df6-88d055beeaa7" containerName="revoke-keys"
Jul 14 01:14:42.101480 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.101463    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.104321 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.104298    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-6969890f-serving-cert\""
Jul 14 01:14:42.104321 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.104317    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:14:42.104443 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.104322    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:14:42.104443 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.104322    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:14:42.110806 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.110786    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"]
Jul 14 01:14:42.200670 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.200639    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.200860 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.200772    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-556dw\" (UniqueName: \"kubernetes.io/projected/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-kube-api-access-556dw\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.200860 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.200820    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.301598 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.301563    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-556dw\" (UniqueName: \"kubernetes.io/projected/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-kube-api-access-556dw\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.301598 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.301601    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.301846 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:14:42.301701    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-serving-cert: secret "maas-api-e2e-migrate-6969890f-serving-cert" not found
Jul 14 01:14:42.301846 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.301728    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.301846 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:14:42.301761    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls podName:0e9d3acf-68fb-47b6-91da-7ef17abbc3d0 nodeName:}" failed. No retries permitted until 2026-07-14 01:14:42.801744454 +0000 UTC m=+1832.782829093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls") pod "maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" (UID: "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0") : secret "maas-api-e2e-migrate-6969890f-serving-cert" not found
Jul 14 01:14:42.302333 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.302311    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-openshift-service-ca\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.312487 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.312461    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-556dw\" (UniqueName: \"kubernetes.io/projected/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-kube-api-access-556dw\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.807039 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:42.807011    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:42.807194 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:14:42.807168    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-serving-cert: secret "maas-api-e2e-migrate-6969890f-serving-cert" not found
Jul 14 01:14:42.807250 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:14:42.807239    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls podName:0e9d3acf-68fb-47b6-91da-7ef17abbc3d0 nodeName:}" failed. No retries permitted until 2026-07-14 01:14:43.807225326 +0000 UTC m=+1833.788309966 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls") pod "maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" (UID: "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0") : secret "maas-api-e2e-migrate-6969890f-serving-cert" not found
Jul 14 01:14:43.815825 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:43.815792    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:43.817988 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:43.817970    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls\") pod \"maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:43.911839 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:43.911795    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:44.057669 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:44.057645    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"]
Jul 14 01:14:44.059762 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:14:44.059732    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e9d3acf_68fb_47b6_91da_7ef17abbc3d0.slice/crio-d84a52f963f3b3f1227a1d118ca60d4bcaaed17a53f21285ca81d9822265d33c WatchSource:0}: Error finding container d84a52f963f3b3f1227a1d118ca60d4bcaaed17a53f21285ca81d9822265d33c: Status 404 returned error can't find the container with id d84a52f963f3b3f1227a1d118ca60d4bcaaed17a53f21285ca81d9822265d33c
Jul 14 01:14:44.098455 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:44.098423    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" event={"ID":"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0","Type":"ContainerStarted","Data":"d84a52f963f3b3f1227a1d118ca60d4bcaaed17a53f21285ca81d9822265d33c"}
Jul 14 01:14:45.104214 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:45.104185    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" event={"ID":"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0","Type":"ContainerStarted","Data":"4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484"}
Jul 14 01:14:45.104549 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:45.104316    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:14:45.120971 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:45.120914    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" podStartSLOduration=2.154375317 podStartE2EDuration="3.120896815s" totalImagesPullingTime="966.521498ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:14:42 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:14:45.119685484 +0000 UTC m=+1835.100770272" watchObservedRunningTime="2026-07-14 01:14:45.120896815 +0000 UTC m=+1835.101981480"
Jul 14 01:14:46.565888 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.565852    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"]
Jul 14 01:14:46.568311 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.568296    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"
Jul 14 01:14:46.571023 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.570995    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:14:46.575441 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.575250    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"]
Jul 14 01:14:46.640550 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.640519    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd55284-803f-45e0-bf0b-b1663ef1ac7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6969890f-hszgw\" (UID: \"fdd55284-803f-45e0-bf0b-b1663ef1ac7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"
Jul 14 01:14:46.741884 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.741854    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd55284-803f-45e0-bf0b-b1663ef1ac7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6969890f-hszgw\" (UID: \"fdd55284-803f-45e0-bf0b-b1663ef1ac7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"
Jul 14 01:14:46.742473 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.742452    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd55284-803f-45e0-bf0b-b1663ef1ac7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6969890f-hszgw\" (UID: \"fdd55284-803f-45e0-bf0b-b1663ef1ac7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"
Jul 14 01:14:46.879347 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:46.879310    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"
Jul 14 01:14:47.000416 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:47.000384    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"]
Jul 14 01:14:47.003690 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:14:47.003663    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfdd55284_803f_45e0_bf0b_b1663ef1ac7a.slice/crio-b21c7421e35991d7f341bdf4d4756110f3f8015cadac7f3b0ea7be0e2875f22e WatchSource:0}: Error finding container b21c7421e35991d7f341bdf4d4756110f3f8015cadac7f3b0ea7be0e2875f22e: Status 404 returned error can't find the container with id b21c7421e35991d7f341bdf4d4756110f3f8015cadac7f3b0ea7be0e2875f22e
Jul 14 01:14:47.114182 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:47.114149    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw" event={"ID":"fdd55284-803f-45e0-bf0b-b1663ef1ac7a","Type":"ContainerStarted","Data":"b21c7421e35991d7f341bdf4d4756110f3f8015cadac7f3b0ea7be0e2875f22e"}
Jul 14 01:14:48.119733 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:48.119692    2552 generic.go:356] "Generic (PLEG): container finished" podID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerID="7bed6bfc54c13a341ef635df887ac1b81ef7eedf29aa1c99264e427fa5dbb6a0" exitCode=7
Jul 14 01:14:48.120096 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:48.119735    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw" event={"ID":"fdd55284-803f-45e0-bf0b-b1663ef1ac7a","Type":"ContainerDied","Data":"7bed6bfc54c13a341ef635df887ac1b81ef7eedf29aa1c99264e427fa5dbb6a0"}
Jul 14 01:14:48.120096 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:48.119970    2552 scope.go:117] "RemoveContainer" containerID="7bed6bfc54c13a341ef635df887ac1b81ef7eedf29aa1c99264e427fa5dbb6a0"
Jul 14 01:14:48.128673 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:48.128652    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:14:49.126037 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:49.126007    2552 generic.go:356] "Generic (PLEG): container finished" podID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerID="9defc093dac616149c124c8b675cb06869cef422b21fe9b794bd985753c10d75" exitCode=7
Jul 14 01:14:49.126422 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:49.126099    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw" event={"ID":"fdd55284-803f-45e0-bf0b-b1663ef1ac7a","Type":"ContainerDied","Data":"9defc093dac616149c124c8b675cb06869cef422b21fe9b794bd985753c10d75"}
Jul 14 01:14:49.126422 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:49.126170    2552 scope.go:117] "RemoveContainer" containerID="7bed6bfc54c13a341ef635df887ac1b81ef7eedf29aa1c99264e427fa5dbb6a0"
Jul 14 01:14:49.126422 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:49.126326    2552 scope.go:117] "RemoveContainer" containerID="9defc093dac616149c124c8b675cb06869cef422b21fe9b794bd985753c10d75"
Jul 14 01:14:49.126581 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:14:49.126560    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-6969890f-hszgw_odh-ai-gateway-infra(fdd55284-803f-45e0-bf0b-b1663ef1ac7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a"
Jul 14 01:14:50.132190 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:50.132164    2552 scope.go:117] "RemoveContainer" containerID="9defc093dac616149c124c8b675cb06869cef422b21fe9b794bd985753c10d75"
Jul 14 01:14:50.132645 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:14:50.132387    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-migrate-6969890f-hszgw_odh-ai-gateway-infra(fdd55284-803f-45e0-bf0b-b1663ef1ac7a)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a"
Jul 14 01:14:51.114446 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:14:51.114416    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:15:00.132599 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.132557    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95"]
Jul 14 01:15:00.135377 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.135360    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"]
Jul 14 01:15:00.135507 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.135475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95"
Jul 14 01:15:00.137341 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.137326    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"
Jul 14 01:15:00.141547 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.141527    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95"]
Jul 14 01:15:00.145143 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.144546    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"]
Jul 14 01:15:00.149488 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.149469    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95"
Jul 14 01:15:00.154775 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.154758    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"
Jul 14 01:15:00.286448 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.286245    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95"]
Jul 14 01:15:00.287978 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:15:00.287943    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60240235_8aa5_4712_9ca3_30ab4ddb7d4d.slice/crio-53925b5a04950546c1f818e327eda4dfe928baa5c4fa081294b7695d5909d52d WatchSource:0}: Error finding container 53925b5a04950546c1f818e327eda4dfe928baa5c4fa081294b7695d5909d52d: Status 404 returned error can't find the container with id 53925b5a04950546c1f818e327eda4dfe928baa5c4fa081294b7695d5909d52d
Jul 14 01:15:00.304816 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:00.304795    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"]
Jul 14 01:15:00.306597 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:15:00.306571    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76464d26_49e9_4a34_b133_76b6428caf13.slice/crio-dc380744acfacff2f7653c78aa3a24035cf251017efdc46d31bac00280f37f7d WatchSource:0}: Error finding container dc380744acfacff2f7653c78aa3a24035cf251017efdc46d31bac00280f37f7d: Status 404 returned error can't find the container with id dc380744acfacff2f7653c78aa3a24035cf251017efdc46d31bac00280f37f7d
Jul 14 01:15:01.184790 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:01.184751    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9" event={"ID":"76464d26-49e9-4a34-b133-76b6428caf13","Type":"ContainerStarted","Data":"f6a6c52c74760c9bd94ca1aec019d90fc7ba2780640da5cec1b773bba34812d4"}
Jul 14 01:15:01.185227 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:01.184824    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9" event={"ID":"76464d26-49e9-4a34-b133-76b6428caf13","Type":"ContainerStarted","Data":"dc380744acfacff2f7653c78aa3a24035cf251017efdc46d31bac00280f37f7d"}
Jul 14 01:15:01.186225 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:01.186203    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95" event={"ID":"60240235-8aa5-4712-9ca3-30ab4ddb7d4d","Type":"ContainerStarted","Data":"5757f9f05ed151db42dc03e95a2c7ba7a2b3a4cea1852f33e95b93633e537c4f"}
Jul 14 01:15:01.186332 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:01.186241    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95" event={"ID":"60240235-8aa5-4712-9ca3-30ab4ddb7d4d","Type":"ContainerStarted","Data":"53925b5a04950546c1f818e327eda4dfe928baa5c4fa081294b7695d5909d52d"}
Jul 14 01:15:01.200078 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:01.200033    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9" podStartSLOduration=1.200016644 podStartE2EDuration="1.200016644s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:15:01.199702291 +0000 UTC m=+1851.180786951" watchObservedRunningTime="2026-07-14 01:15:01.200016644 +0000 UTC m=+1851.181101306"
Jul 14 01:15:01.214660 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:01.214610    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95" podStartSLOduration=1.214593969 podStartE2EDuration="1.214593969s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:15:01.214450666 +0000 UTC m=+1851.195535330" watchObservedRunningTime="2026-07-14 01:15:01.214593969 +0000 UTC m=+1851.195678633"
Jul 14 01:15:02.191940 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:02.191906    2552 generic.go:356] "Generic (PLEG): container finished" podID="76464d26-49e9-4a34-b133-76b6428caf13" containerID="f6a6c52c74760c9bd94ca1aec019d90fc7ba2780640da5cec1b773bba34812d4" exitCode=0
Jul 14 01:15:02.192335 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:02.191993    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9" event={"ID":"76464d26-49e9-4a34-b133-76b6428caf13","Type":"ContainerDied","Data":"f6a6c52c74760c9bd94ca1aec019d90fc7ba2780640da5cec1b773bba34812d4"}
Jul 14 01:15:02.193425 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:02.193400    2552 generic.go:356] "Generic (PLEG): container finished" podID="60240235-8aa5-4712-9ca3-30ab4ddb7d4d" containerID="5757f9f05ed151db42dc03e95a2c7ba7a2b3a4cea1852f33e95b93633e537c4f" exitCode=0
Jul 14 01:15:02.193524 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:02.193459    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95" event={"ID":"60240235-8aa5-4712-9ca3-30ab4ddb7d4d","Type":"ContainerDied","Data":"5757f9f05ed151db42dc03e95a2c7ba7a2b3a4cea1852f33e95b93633e537c4f"}
Jul 14 01:15:03.325137 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:03.325115    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"
Jul 14 01:15:03.354969 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:03.354947    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95"
Jul 14 01:15:04.208171 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:04.208139    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"
Jul 14 01:15:04.208359 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:04.208166    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9" event={"ID":"76464d26-49e9-4a34-b133-76b6428caf13","Type":"ContainerDied","Data":"dc380744acfacff2f7653c78aa3a24035cf251017efdc46d31bac00280f37f7d"}
Jul 14 01:15:04.208359 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:04.208247    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc380744acfacff2f7653c78aa3a24035cf251017efdc46d31bac00280f37f7d"
Jul 14 01:15:04.209449 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:04.209431    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95"
Jul 14 01:15:04.209560 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:04.209465    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29733195-qsg95" event={"ID":"60240235-8aa5-4712-9ca3-30ab4ddb7d4d","Type":"ContainerDied","Data":"53925b5a04950546c1f818e327eda4dfe928baa5c4fa081294b7695d5909d52d"}
Jul 14 01:15:04.209560 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:04.209521    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53925b5a04950546c1f818e327eda4dfe928baa5c4fa081294b7695d5909d52d"
Jul 14 01:15:05.518692 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:05.518668    2552 scope.go:117] "RemoveContainer" containerID="9defc093dac616149c124c8b675cb06869cef422b21fe9b794bd985753c10d75"
Jul 14 01:15:06.221588 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:06.221558    2552 generic.go:356] "Generic (PLEG): container finished" podID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerID="2f3b6e5c9f6d1096b86f9474ddc6f619bf53f7a22aeebacd4bae0b690a8a761b" exitCode=0
Jul 14 01:15:06.221753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:06.221632    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw" event={"ID":"fdd55284-803f-45e0-bf0b-b1663ef1ac7a","Type":"ContainerDied","Data":"2f3b6e5c9f6d1096b86f9474ddc6f619bf53f7a22aeebacd4bae0b690a8a761b"}
Jul 14 01:15:06.221753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:06.221703    2552 scope.go:117] "RemoveContainer" containerID="9defc093dac616149c124c8b675cb06869cef422b21fe9b794bd985753c10d75"
Jul 14 01:15:07.251844 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:07.251813    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"]
Jul 14 01:15:07.356557 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:07.356536    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"
Jul 14 01:15:07.419111 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:07.419083    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd55284-803f-45e0-bf0b-b1663ef1ac7a-maas-api-service-ca\") pod \"fdd55284-803f-45e0-bf0b-b1663ef1ac7a\" (UID: \"fdd55284-803f-45e0-bf0b-b1663ef1ac7a\") "
Jul 14 01:15:07.419435 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:07.419412    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fdd55284-803f-45e0-bf0b-b1663ef1ac7a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fdd55284-803f-45e0-bf0b-b1663ef1ac7a" (UID: "fdd55284-803f-45e0-bf0b-b1663ef1ac7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:15:07.519855 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:07.519763    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fdd55284-803f-45e0-bf0b-b1663ef1ac7a-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:15:08.234458 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:08.234417    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw" event={"ID":"fdd55284-803f-45e0-bf0b-b1663ef1ac7a","Type":"ContainerDied","Data":"b21c7421e35991d7f341bdf4d4756110f3f8015cadac7f3b0ea7be0e2875f22e"}
Jul 14 01:15:08.234643 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:08.234520    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b21c7421e35991d7f341bdf4d4756110f3f8015cadac7f3b0ea7be0e2875f22e"
Jul 14 01:15:08.234643 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:08.234435    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"
Jul 14 01:15:08.261866 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:08.261838    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"]
Jul 14 01:15:08.265485 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:08.265464    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-hszgw"]
Jul 14 01:15:08.524119 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:08.524042    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" path="/var/lib/kubelet/pods/fdd55284-803f-45e0-bf0b-b1663ef1ac7a/volumes"
Jul 14 01:15:46.618604 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618527    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"]
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618871    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618883    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618900    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618911    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618924    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60240235-8aa5-4712-9ca3-30ab4ddb7d4d" containerName="cleanup"
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618930    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="60240235-8aa5-4712-9ca3-30ab4ddb7d4d" containerName="cleanup"
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618936    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76464d26-49e9-4a34-b133-76b6428caf13" containerName="cleanup"
Jul 14 01:15:46.618993 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.618941    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="76464d26-49e9-4a34-b133-76b6428caf13" containerName="cleanup"
Jul 14 01:15:46.619222 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.619001    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="60240235-8aa5-4712-9ca3-30ab4ddb7d4d" containerName="cleanup"
Jul 14 01:15:46.619222 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.619010    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:46.619222 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.619015    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="76464d26-49e9-4a34-b133-76b6428caf13" containerName="cleanup"
Jul 14 01:15:46.619222 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.619022    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:46.620968 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.620953    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"
Jul 14 01:15:46.623372 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.623352    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:15:46.629424 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.629399    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"]
Jul 14 01:15:46.731923 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.731886    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7fd887-bd80-4198-b758-ca59fbdc9550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q\" (UID: \"3a7fd887-bd80-4198-b758-ca59fbdc9550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"
Jul 14 01:15:46.833142 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.833094    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7fd887-bd80-4198-b758-ca59fbdc9550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q\" (UID: \"3a7fd887-bd80-4198-b758-ca59fbdc9550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"
Jul 14 01:15:46.833753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.833735    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7fd887-bd80-4198-b758-ca59fbdc9550-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q\" (UID: \"3a7fd887-bd80-4198-b758-ca59fbdc9550\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"
Jul 14 01:15:46.931795 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:46.931714    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"
Jul 14 01:15:47.062711 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:47.062684    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"]
Jul 14 01:15:47.064477 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:15:47.064448    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a7fd887_bd80_4198_b758_ca59fbdc9550.slice/crio-d6c6a29cabf454b02bb34d9abd973477d920f076ee3e61b84c42a556084ca44e WatchSource:0}: Error finding container d6c6a29cabf454b02bb34d9abd973477d920f076ee3e61b84c42a556084ca44e: Status 404 returned error can't find the container with id d6c6a29cabf454b02bb34d9abd973477d920f076ee3e61b84c42a556084ca44e
Jul 14 01:15:47.423040 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:47.423007    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q" event={"ID":"3a7fd887-bd80-4198-b758-ca59fbdc9550","Type":"ContainerStarted","Data":"c942ffaf73201741b6794e914b5649438bcf1cdf2ea8f8b945108c699c33aace"}
Jul 14 01:15:47.423211 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:47.423057    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q" event={"ID":"3a7fd887-bd80-4198-b758-ca59fbdc9550","Type":"ContainerStarted","Data":"d6c6a29cabf454b02bb34d9abd973477d920f076ee3e61b84c42a556084ca44e"}
Jul 14 01:15:47.438703 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:47.438649    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q" podStartSLOduration=1.438634475 podStartE2EDuration="1.438634475s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:15:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:15:47.436770276 +0000 UTC m=+1897.417854938" watchObservedRunningTime="2026-07-14 01:15:47.438634475 +0000 UTC m=+1897.419719152"
Jul 14 01:15:48.429101 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:48.429063    2552 generic.go:356] "Generic (PLEG): container finished" podID="3a7fd887-bd80-4198-b758-ca59fbdc9550" containerID="c942ffaf73201741b6794e914b5649438bcf1cdf2ea8f8b945108c699c33aace" exitCode=0
Jul 14 01:15:48.429490 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:48.429154    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q" event={"ID":"3a7fd887-bd80-4198-b758-ca59fbdc9550","Type":"ContainerDied","Data":"c942ffaf73201741b6794e914b5649438bcf1cdf2ea8f8b945108c699c33aace"}
Jul 14 01:15:49.426674 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:49.426636    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:15:49.561147 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:49.561127    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"
Jul 14 01:15:49.655746 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:49.655715    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7fd887-bd80-4198-b758-ca59fbdc9550-maas-api-service-ca\") pod \"3a7fd887-bd80-4198-b758-ca59fbdc9550\" (UID: \"3a7fd887-bd80-4198-b758-ca59fbdc9550\") "
Jul 14 01:15:49.656073 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:49.656048    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3a7fd887-bd80-4198-b758-ca59fbdc9550-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "3a7fd887-bd80-4198-b758-ca59fbdc9550" (UID: "3a7fd887-bd80-4198-b758-ca59fbdc9550"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:15:49.757225 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:49.757142    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3a7fd887-bd80-4198-b758-ca59fbdc9550-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:15:50.440241 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:50.440209    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q" event={"ID":"3a7fd887-bd80-4198-b758-ca59fbdc9550","Type":"ContainerDied","Data":"d6c6a29cabf454b02bb34d9abd973477d920f076ee3e61b84c42a556084ca44e"}
Jul 14 01:15:50.440411 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:50.440256    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6c6a29cabf454b02bb34d9abd973477d920f076ee3e61b84c42a556084ca44e"
Jul 14 01:15:50.440411 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:50.440221    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q"
Jul 14 01:15:51.858985 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:51.858947    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"]
Jul 14 01:15:51.859476 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:51.859198    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" podUID="0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" containerName="maas-api" containerID="cri-o://4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484" gracePeriod=30
Jul 14 01:15:51.883445 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:51.883421    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"]
Jul 14 01:15:51.885249 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:51.885230    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-migrate-6969890f-29733195-w2fw9"]
Jul 14 01:15:52.099651 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.099628    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:15:52.182257 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.182177    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls\") pod \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") "
Jul 14 01:15:52.182257 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.182225    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-openshift-service-ca\") pod \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") "
Jul 14 01:15:52.182459 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.182318    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-556dw\" (UniqueName: \"kubernetes.io/projected/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-kube-api-access-556dw\") pod \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\" (UID: \"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0\") "
Jul 14 01:15:52.182628 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.182603    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" (UID: "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:15:52.184284 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.184263    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" (UID: "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:15:52.184364 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.184286    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-kube-api-access-556dw" (OuterVolumeSpecName: "kube-api-access-556dw") pod "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" (UID: "0e9d3acf-68fb-47b6-91da-7ef17abbc3d0"). InnerVolumeSpecName "kube-api-access-556dw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:15:52.283101 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.283049    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-556dw\" (UniqueName: \"kubernetes.io/projected/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-kube-api-access-556dw\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:15:52.283101 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.283095    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:15:52.283101 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.283111    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:15:52.449397 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.449316    2552 generic.go:356] "Generic (PLEG): container finished" podID="0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" containerID="4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484" exitCode=0
Jul 14 01:15:52.449397 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.449354    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" event={"ID":"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0","Type":"ContainerDied","Data":"4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484"}
Jul 14 01:15:52.449397 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.449368    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"
Jul 14 01:15:52.449639 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.449418    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6" event={"ID":"0e9d3acf-68fb-47b6-91da-7ef17abbc3d0","Type":"ContainerDied","Data":"d84a52f963f3b3f1227a1d118ca60d4bcaaed17a53f21285ca81d9822265d33c"}
Jul 14 01:15:52.449639 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.449439    2552 scope.go:117] "RemoveContainer" containerID="4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484"
Jul 14 01:15:52.458300 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.458284    2552 scope.go:117] "RemoveContainer" containerID="4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484"
Jul 14 01:15:52.458539 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:15:52.458519    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484\": container with ID starting with 4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484 not found: ID does not exist" containerID="4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484"
Jul 14 01:15:52.458603 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.458548    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484"} err="failed to get container status \"4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484\": rpc error: code = NotFound desc = could not find container \"4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484\": container with ID starting with 4be52d707a861b6f0a68e0983a69f381dafdfbaad8ea289662cb5e06a5c78484 not found: ID does not exist"
Jul 14 01:15:52.470058 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.470026    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"]
Jul 14 01:15:52.471866 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.471845    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-6969890f-6d947568dd-7s9g6"]
Jul 14 01:15:52.523888 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.523851    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" path="/var/lib/kubelet/pods/0e9d3acf-68fb-47b6-91da-7ef17abbc3d0/volumes"
Jul 14 01:15:52.524161 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:52.524146    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76464d26-49e9-4a34-b133-76b6428caf13" path="/var/lib/kubelet/pods/76464d26-49e9-4a34-b133-76b6428caf13/volumes"
Jul 14 01:15:59.685838 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.685808    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"]
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686134    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" containerName="maas-api"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686144    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" containerName="maas-api"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686154    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686159    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686165    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a7fd887-bd80-4198-b758-ca59fbdc9550" containerName="revoke-keys"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686170    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a7fd887-bd80-4198-b758-ca59fbdc9550" containerName="revoke-keys"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686220    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e9d3acf-68fb-47b6-91da-7ef17abbc3d0" containerName="maas-api"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686227    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fdd55284-803f-45e0-bf0b-b1663ef1ac7a" containerName="revoke-keys"
Jul 14 01:15:59.686282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.686236    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a7fd887-bd80-4198-b758-ca59fbdc9550" containerName="revoke-keys"
Jul 14 01:15:59.698398 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.698364    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"]
Jul 14 01:15:59.698596 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.698486    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.701120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.701091    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:15:59.701282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.701104    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-f0fe996a-serving-cert\""
Jul 14 01:15:59.701396 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.701171    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:15:59.701454 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.701235    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:15:59.847764 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.847730    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01ce092-a302-4a98-a6e4-9497673dcdb7-maas-api-tls\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.847764 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.847771    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01ce092-a302-4a98-a6e4-9497673dcdb7-openshift-service-ca\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.847992 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.847864    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjv4p\" (UniqueName: \"kubernetes.io/projected/d01ce092-a302-4a98-a6e4-9497673dcdb7-kube-api-access-mjv4p\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.948617 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.948521    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01ce092-a302-4a98-a6e4-9497673dcdb7-maas-api-tls\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.948617 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.948559    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01ce092-a302-4a98-a6e4-9497673dcdb7-openshift-service-ca\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.948617 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.948619    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjv4p\" (UniqueName: \"kubernetes.io/projected/d01ce092-a302-4a98-a6e4-9497673dcdb7-kube-api-access-mjv4p\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.949435 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.949410    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01ce092-a302-4a98-a6e4-9497673dcdb7-openshift-service-ca\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.951116 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.951096    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01ce092-a302-4a98-a6e4-9497673dcdb7-maas-api-tls\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:15:59.956630 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:15:59.956602    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjv4p\" (UniqueName: \"kubernetes.io/projected/d01ce092-a302-4a98-a6e4-9497673dcdb7-kube-api-access-mjv4p\") pod \"maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:16:00.009417 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:00.009387    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:16:00.137392 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:00.137339    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"]
Jul 14 01:16:00.139773 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:16:00.139745    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd01ce092_a302_4a98_a6e4_9497673dcdb7.slice/crio-3c5756680517a5192be92f394de46a7d0ee627845c5a0adcb8fb1b7299d1e604 WatchSource:0}: Error finding container 3c5756680517a5192be92f394de46a7d0ee627845c5a0adcb8fb1b7299d1e604: Status 404 returned error can't find the container with id 3c5756680517a5192be92f394de46a7d0ee627845c5a0adcb8fb1b7299d1e604
Jul 14 01:16:00.488737 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:00.488702    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84" event={"ID":"d01ce092-a302-4a98-a6e4-9497673dcdb7","Type":"ContainerStarted","Data":"3c5756680517a5192be92f394de46a7d0ee627845c5a0adcb8fb1b7299d1e604"}
Jul 14 01:16:01.494963 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:01.494931    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84" event={"ID":"d01ce092-a302-4a98-a6e4-9497673dcdb7","Type":"ContainerStarted","Data":"b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8"}
Jul 14 01:16:01.495313 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:01.495018    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:16:01.515226 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:01.515179    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84" podStartSLOduration=1.9290326549999999 podStartE2EDuration="2.515166075s" totalImagesPullingTime="586.13342ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:16:01.513189654 +0000 UTC m=+1911.494274337" watchObservedRunningTime="2026-07-14 01:16:01.515166075 +0000 UTC m=+1911.496250735"
Jul 14 01:16:02.626121 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:02.626084    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:16:04.776917 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:04.776879    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"]
Jul 14 01:16:04.779741 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:04.779725    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"
Jul 14 01:16:04.782379 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:04.782362    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:16:04.787895 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:04.787856    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"]
Jul 14 01:16:04.893523 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:04.893461    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9b13dc-eccf-4446-95a4-43204676ef87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5\" (UID: \"cc9b13dc-eccf-4446-95a4-43204676ef87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"
Jul 14 01:16:04.994820 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:04.994779    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9b13dc-eccf-4446-95a4-43204676ef87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5\" (UID: \"cc9b13dc-eccf-4446-95a4-43204676ef87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"
Jul 14 01:16:04.995390 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:04.995369    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9b13dc-eccf-4446-95a4-43204676ef87-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5\" (UID: \"cc9b13dc-eccf-4446-95a4-43204676ef87\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"
Jul 14 01:16:05.092210 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:05.092183    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"
Jul 14 01:16:05.231677 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:05.231655    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"]
Jul 14 01:16:05.233873 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:16:05.233844    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc9b13dc_eccf_4446_95a4_43204676ef87.slice/crio-02c5f1df4962581fe3fdcd6c131acc00eb1565c0df273e9b97c172eae2d20255 WatchSource:0}: Error finding container 02c5f1df4962581fe3fdcd6c131acc00eb1565c0df273e9b97c172eae2d20255: Status 404 returned error can't find the container with id 02c5f1df4962581fe3fdcd6c131acc00eb1565c0df273e9b97c172eae2d20255
Jul 14 01:16:05.514999 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:05.514964    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" event={"ID":"cc9b13dc-eccf-4446-95a4-43204676ef87","Type":"ContainerStarted","Data":"265c360f045b37b80cf9259c5402eebc316ce56c24654372483e3c9f279667b2"}
Jul 14 01:16:05.515163 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:05.515013    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" event={"ID":"cc9b13dc-eccf-4446-95a4-43204676ef87","Type":"ContainerStarted","Data":"02c5f1df4962581fe3fdcd6c131acc00eb1565c0df273e9b97c172eae2d20255"}
Jul 14 01:16:05.532120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:05.532066    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" podStartSLOduration=1.532051286 podStartE2EDuration="1.532051286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:16:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:16:05.529306072 +0000 UTC m=+1915.510390735" watchObservedRunningTime="2026-07-14 01:16:05.532051286 +0000 UTC m=+1915.513135946"
Jul 14 01:16:06.521678 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:06.521639    2552 generic.go:356] "Generic (PLEG): container finished" podID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerID="265c360f045b37b80cf9259c5402eebc316ce56c24654372483e3c9f279667b2" exitCode=7
Jul 14 01:16:06.522169 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:06.521883    2552 scope.go:117] "RemoveContainer" containerID="265c360f045b37b80cf9259c5402eebc316ce56c24654372483e3c9f279667b2"
Jul 14 01:16:06.525039 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:06.525014    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" event={"ID":"cc9b13dc-eccf-4446-95a4-43204676ef87","Type":"ContainerDied","Data":"265c360f045b37b80cf9259c5402eebc316ce56c24654372483e3c9f279667b2"}
Jul 14 01:16:07.506270 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:07.506196    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:16:07.528755 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:07.528721    2552 generic.go:356] "Generic (PLEG): container finished" podID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerID="ff5bc6fa223e14b63c4de866ee78108a487faceee26c9ccaa42c83d7a5af1857" exitCode=7
Jul 14 01:16:07.528755 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:07.528756    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" event={"ID":"cc9b13dc-eccf-4446-95a4-43204676ef87","Type":"ContainerDied","Data":"ff5bc6fa223e14b63c4de866ee78108a487faceee26c9ccaa42c83d7a5af1857"}
Jul 14 01:16:07.529247 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:07.528798    2552 scope.go:117] "RemoveContainer" containerID="265c360f045b37b80cf9259c5402eebc316ce56c24654372483e3c9f279667b2"
Jul 14 01:16:07.529247 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:07.529042    2552 scope.go:117] "RemoveContainer" containerID="ff5bc6fa223e14b63c4de866ee78108a487faceee26c9ccaa42c83d7a5af1857"
Jul 14 01:16:07.529338 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:16:07.529258    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5_odh-ai-gateway-infra(cc9b13dc-eccf-4446-95a4-43204676ef87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87"
Jul 14 01:16:08.535174 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:08.535149    2552 scope.go:117] "RemoveContainer" containerID="ff5bc6fa223e14b63c4de866ee78108a487faceee26c9ccaa42c83d7a5af1857"
Jul 14 01:16:08.535601 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:16:08.535343    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5_odh-ai-gateway-infra(cc9b13dc-eccf-4446-95a4-43204676ef87)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87"
Jul 14 01:16:21.518596 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:21.518564    2552 scope.go:117] "RemoveContainer" containerID="ff5bc6fa223e14b63c4de866ee78108a487faceee26c9ccaa42c83d7a5af1857"
Jul 14 01:16:22.604828 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:22.604790    2552 generic.go:356] "Generic (PLEG): container finished" podID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerID="a7c8d4f2456698ba03aaf4aefee3ce687e38e5344046261bb2270ff3e76bbaa2" exitCode=0
Jul 14 01:16:22.605226 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:22.604863    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" event={"ID":"cc9b13dc-eccf-4446-95a4-43204676ef87","Type":"ContainerDied","Data":"a7c8d4f2456698ba03aaf4aefee3ce687e38e5344046261bb2270ff3e76bbaa2"}
Jul 14 01:16:22.605226 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:22.604920    2552 scope.go:117] "RemoveContainer" containerID="ff5bc6fa223e14b63c4de866ee78108a487faceee26c9ccaa42c83d7a5af1857"
Jul 14 01:16:23.634859 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:23.634826    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"]
Jul 14 01:16:23.734350 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:23.734324    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"
Jul 14 01:16:23.842622 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:23.842593    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9b13dc-eccf-4446-95a4-43204676ef87-maas-api-service-ca\") pod \"cc9b13dc-eccf-4446-95a4-43204676ef87\" (UID: \"cc9b13dc-eccf-4446-95a4-43204676ef87\") "
Jul 14 01:16:23.842972 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:23.842951    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc9b13dc-eccf-4446-95a4-43204676ef87-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cc9b13dc-eccf-4446-95a4-43204676ef87" (UID: "cc9b13dc-eccf-4446-95a4-43204676ef87"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:16:23.943840 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:23.943760    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cc9b13dc-eccf-4446-95a4-43204676ef87-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:16:24.617535 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:24.617491    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"
Jul 14 01:16:24.617535 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:24.617520    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5" event={"ID":"cc9b13dc-eccf-4446-95a4-43204676ef87","Type":"ContainerDied","Data":"02c5f1df4962581fe3fdcd6c131acc00eb1565c0df273e9b97c172eae2d20255"}
Jul 14 01:16:24.617725 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:24.617569    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02c5f1df4962581fe3fdcd6c131acc00eb1565c0df273e9b97c172eae2d20255"
Jul 14 01:16:24.646381 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:24.646350    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"]
Jul 14 01:16:24.649611 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:24.649588    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-zdtp5"]
Jul 14 01:16:26.523579 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:26.523547    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" path="/var/lib/kubelet/pods/cc9b13dc-eccf-4446-95a4-43204676ef87/volumes"
Jul 14 01:16:54.819017 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.818981    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"]
Jul 14 01:16:54.819484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.819313    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:16:54.819484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.819324    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:16:54.819484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.819334    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:16:54.819484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.819339    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:16:54.819484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.819405    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:16:54.819484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.819411    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:16:54.819484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.819418    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:16:54.822646 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.822627    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"
Jul 14 01:16:54.825413 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.825386    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:16:54.830304 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.830278    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"]
Jul 14 01:16:54.923711 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:54.923661    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj\" (UID: \"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"
Jul 14 01:16:55.024646 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:55.024606    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj\" (UID: \"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"
Jul 14 01:16:55.025231 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:55.025211    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj\" (UID: \"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"
Jul 14 01:16:55.133031 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:55.132933    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"
Jul 14 01:16:55.259579 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:55.259552    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"]
Jul 14 01:16:55.262158 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:16:55.262130    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb3a8b3ac_a1ee_4f32_b2bc_0dfdd57dfdc4.slice/crio-b8d888aa6672195d5d5e82544457155dc09f7eaade4111e869bb77327cc5e8c3 WatchSource:0}: Error finding container b8d888aa6672195d5d5e82544457155dc09f7eaade4111e869bb77327cc5e8c3: Status 404 returned error can't find the container with id b8d888aa6672195d5d5e82544457155dc09f7eaade4111e869bb77327cc5e8c3
Jul 14 01:16:55.762758 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:55.762664    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj" event={"ID":"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4","Type":"ContainerStarted","Data":"421e173e783701fbafd3ec976b98bf7f4d2a3b0e98d48ab67184131faebddb66"}
Jul 14 01:16:55.762758 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:55.762728    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj" event={"ID":"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4","Type":"ContainerStarted","Data":"b8d888aa6672195d5d5e82544457155dc09f7eaade4111e869bb77327cc5e8c3"}
Jul 14 01:16:55.780223 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:55.780172    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj" podStartSLOduration=1.780154414 podStartE2EDuration="1.780154414s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:16:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:16:55.777633894 +0000 UTC m=+1965.758718554" watchObservedRunningTime="2026-07-14 01:16:55.780154414 +0000 UTC m=+1965.761239074"
Jul 14 01:16:56.768951 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:56.768920    2552 generic.go:356] "Generic (PLEG): container finished" podID="b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4" containerID="421e173e783701fbafd3ec976b98bf7f4d2a3b0e98d48ab67184131faebddb66" exitCode=0
Jul 14 01:16:56.769329 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:56.768973    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj" event={"ID":"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4","Type":"ContainerDied","Data":"421e173e783701fbafd3ec976b98bf7f4d2a3b0e98d48ab67184131faebddb66"}
Jul 14 01:16:57.897693 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:57.897666    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"
Jul 14 01:16:58.051567 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:58.051452    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4-maas-api-service-ca\") pod \"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4\" (UID: \"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4\") "
Jul 14 01:16:58.051836 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:58.051814    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4" (UID: "b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:16:58.152207 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:58.152173    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:16:58.779300 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:58.779271    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj"
Jul 14 01:16:58.779461 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:58.779308    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj" event={"ID":"b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4","Type":"ContainerDied","Data":"b8d888aa6672195d5d5e82544457155dc09f7eaade4111e869bb77327cc5e8c3"}
Jul 14 01:16:58.779461 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:16:58.779354    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8d888aa6672195d5d5e82544457155dc09f7eaade4111e869bb77327cc5e8c3"
Jul 14 01:17:05.218686 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.218603    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"]
Jul 14 01:17:05.219088 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.218847    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84" podUID="d01ce092-a302-4a98-a6e4-9497673dcdb7" containerName="maas-api" containerID="cri-o://b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8" gracePeriod=30
Jul 14 01:17:05.463089 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.463063    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:17:05.506982 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.506918    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01ce092-a302-4a98-a6e4-9497673dcdb7-openshift-service-ca\") pod \"d01ce092-a302-4a98-a6e4-9497673dcdb7\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") "
Jul 14 01:17:05.506982 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.506964    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mjv4p\" (UniqueName: \"kubernetes.io/projected/d01ce092-a302-4a98-a6e4-9497673dcdb7-kube-api-access-mjv4p\") pod \"d01ce092-a302-4a98-a6e4-9497673dcdb7\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") "
Jul 14 01:17:05.507204 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.507000    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01ce092-a302-4a98-a6e4-9497673dcdb7-maas-api-tls\") pod \"d01ce092-a302-4a98-a6e4-9497673dcdb7\" (UID: \"d01ce092-a302-4a98-a6e4-9497673dcdb7\") "
Jul 14 01:17:05.507319 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.507301    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d01ce092-a302-4a98-a6e4-9497673dcdb7-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d01ce092-a302-4a98-a6e4-9497673dcdb7" (UID: "d01ce092-a302-4a98-a6e4-9497673dcdb7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:17:05.509008 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.508985    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d01ce092-a302-4a98-a6e4-9497673dcdb7-kube-api-access-mjv4p" (OuterVolumeSpecName: "kube-api-access-mjv4p") pod "d01ce092-a302-4a98-a6e4-9497673dcdb7" (UID: "d01ce092-a302-4a98-a6e4-9497673dcdb7"). InnerVolumeSpecName "kube-api-access-mjv4p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:17:05.509166 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.509147    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d01ce092-a302-4a98-a6e4-9497673dcdb7-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d01ce092-a302-4a98-a6e4-9497673dcdb7" (UID: "d01ce092-a302-4a98-a6e4-9497673dcdb7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:17:05.608308 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.608256    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d01ce092-a302-4a98-a6e4-9497673dcdb7-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:17:05.608547 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.608336    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d01ce092-a302-4a98-a6e4-9497673dcdb7-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:17:05.608547 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.608378    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjv4p\" (UniqueName: \"kubernetes.io/projected/d01ce092-a302-4a98-a6e4-9497673dcdb7-kube-api-access-mjv4p\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:17:05.813156 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.813063    2552 generic.go:356] "Generic (PLEG): container finished" podID="d01ce092-a302-4a98-a6e4-9497673dcdb7" containerID="b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8" exitCode=0
Jul 14 01:17:05.813156 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.813129    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"
Jul 14 01:17:05.813351 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.813127    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84" event={"ID":"d01ce092-a302-4a98-a6e4-9497673dcdb7","Type":"ContainerDied","Data":"b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8"}
Jul 14 01:17:05.813351 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.813250    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84" event={"ID":"d01ce092-a302-4a98-a6e4-9497673dcdb7","Type":"ContainerDied","Data":"3c5756680517a5192be92f394de46a7d0ee627845c5a0adcb8fb1b7299d1e604"}
Jul 14 01:17:05.813351 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.813280    2552 scope.go:117] "RemoveContainer" containerID="b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8"
Jul 14 01:17:05.822195 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.822182    2552 scope.go:117] "RemoveContainer" containerID="b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8"
Jul 14 01:17:05.822435 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:17:05.822416    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8\": container with ID starting with b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8 not found: ID does not exist" containerID="b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8"
Jul 14 01:17:05.822485 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.822445    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8"} err="failed to get container status \"b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8\": rpc error: code = NotFound desc = could not find container \"b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8\": container with ID starting with b542eed60fd3b79c9543c4417f307dbe641100f134c7757c1234be9a4106e0b8 not found: ID does not exist"
Jul 14 01:17:05.836168 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.836143    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"]
Jul 14 01:17:05.839746 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:05.839727    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-f0fe996a-7f4bf6b874-ptt84"]
Jul 14 01:17:06.523890 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:06.523848    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d01ce092-a302-4a98-a6e4-9497673dcdb7" path="/var/lib/kubelet/pods/d01ce092-a302-4a98-a6e4-9497673dcdb7/volumes"
Jul 14 01:17:08.428702 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:08.428663    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:17:22.590039 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590008    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"]
Jul 14 01:17:22.590401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590356    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:17:22.590401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590370    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="cc9b13dc-eccf-4446-95a4-43204676ef87" containerName="revoke-keys"
Jul 14 01:17:22.590401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590381    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d01ce092-a302-4a98-a6e4-9497673dcdb7" containerName="maas-api"
Jul 14 01:17:22.590401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590386    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d01ce092-a302-4a98-a6e4-9497673dcdb7" containerName="maas-api"
Jul 14 01:17:22.590401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590397    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4" containerName="revoke-keys"
Jul 14 01:17:22.590401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590403    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4" containerName="revoke-keys"
Jul 14 01:17:22.590615 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590475    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4" containerName="revoke-keys"
Jul 14 01:17:22.590615 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.590485    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d01ce092-a302-4a98-a6e4-9497673dcdb7" containerName="maas-api"
Jul 14 01:17:22.593366 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.593346    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.596464 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.596441    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-b3a22390-serving-cert\""
Jul 14 01:17:22.596614 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.596472    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:17:22.597605 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.597582    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:17:22.597605 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.597596    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:17:22.604987 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.604965    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"]
Jul 14 01:17:22.640478 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.640446    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s97f5\" (UniqueName: \"kubernetes.io/projected/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-kube-api-access-s97f5\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.640648 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.640517    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.640648 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.640563    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-openshift-service-ca\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.741746 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.741699    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s97f5\" (UniqueName: \"kubernetes.io/projected/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-kube-api-access-s97f5\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.741746 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.741759    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.741963 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.741784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-openshift-service-ca\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.741963 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:17:22.741909    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-serving-cert: secret "maas-api-e2e-derive-b3a22390-serving-cert" not found
Jul 14 01:17:22.742048 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:17:22.741989    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls podName:751fa4b5-b0ce-45ab-b20e-0c12bd56e641 nodeName:}" failed. No retries permitted until 2026-07-14 01:17:23.241970169 +0000 UTC m=+1993.223054818 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls") pod "maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf" (UID: "751fa4b5-b0ce-45ab-b20e-0c12bd56e641") : secret "maas-api-e2e-derive-b3a22390-serving-cert" not found
Jul 14 01:17:22.742355 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.742339    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-openshift-service-ca\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:22.750412 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:22.750378    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s97f5\" (UniqueName: \"kubernetes.io/projected/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-kube-api-access-s97f5\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:23.245688 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:23.245652    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:23.248313 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:23.248286    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls\") pod \"maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:23.504562 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:23.504461    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:23.632486 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:23.632419    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"]
Jul 14 01:17:23.634771 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:17:23.634742    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod751fa4b5_b0ce_45ab_b20e_0c12bd56e641.slice/crio-6f9a81e056dd4b59067f876ccaa82bc5e9fc1e0a8d46d1f67fedca5afbd99478 WatchSource:0}: Error finding container 6f9a81e056dd4b59067f876ccaa82bc5e9fc1e0a8d46d1f67fedca5afbd99478: Status 404 returned error can't find the container with id 6f9a81e056dd4b59067f876ccaa82bc5e9fc1e0a8d46d1f67fedca5afbd99478
Jul 14 01:17:23.900454 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:23.900418    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf" event={"ID":"751fa4b5-b0ce-45ab-b20e-0c12bd56e641","Type":"ContainerStarted","Data":"6f9a81e056dd4b59067f876ccaa82bc5e9fc1e0a8d46d1f67fedca5afbd99478"}
Jul 14 01:17:24.829513 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:24.829473    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:17:24.911698 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:24.911666    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf" event={"ID":"751fa4b5-b0ce-45ab-b20e-0c12bd56e641","Type":"ContainerStarted","Data":"d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307"}
Jul 14 01:17:24.911860 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:24.911766    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:24.930050 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:24.930007    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf" podStartSLOduration=2.580385564 podStartE2EDuration="2.929992061s" totalImagesPullingTime="349.606497ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:17:24.927798016 +0000 UTC m=+1994.908882671" watchObservedRunningTime="2026-07-14 01:17:24.929992061 +0000 UTC m=+1994.911076725"
Jul 14 01:17:26.826987 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:26.826950    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"]
Jul 14 01:17:26.830234 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:26.830216    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"
Jul 14 01:17:26.832649 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:26.832629    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:17:26.836685 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:26.836663    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"]
Jul 14 01:17:26.877041 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:26.876996    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae131-0e3b-4bd5-b19a-ee23801580e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3a22390-ftklv\" (UID: \"ef8ae131-0e3b-4bd5-b19a-ee23801580e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"
Jul 14 01:17:26.977992 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:26.977948    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae131-0e3b-4bd5-b19a-ee23801580e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3a22390-ftklv\" (UID: \"ef8ae131-0e3b-4bd5-b19a-ee23801580e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"
Jul 14 01:17:26.978664 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:26.978644    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae131-0e3b-4bd5-b19a-ee23801580e2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3a22390-ftklv\" (UID: \"ef8ae131-0e3b-4bd5-b19a-ee23801580e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"
Jul 14 01:17:27.141973 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:27.141890    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"
Jul 14 01:17:27.273163 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:27.273134    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"]
Jul 14 01:17:27.274929 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:17:27.274899    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef8ae131_0e3b_4bd5_b19a_ee23801580e2.slice/crio-0d82c949438936c7194331ffbc5468d0b3aa464dfcc2d47737a41271c9a76571 WatchSource:0}: Error finding container 0d82c949438936c7194331ffbc5468d0b3aa464dfcc2d47737a41271c9a76571: Status 404 returned error can't find the container with id 0d82c949438936c7194331ffbc5468d0b3aa464dfcc2d47737a41271c9a76571
Jul 14 01:17:27.927814 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:27.927780    2552 generic.go:356] "Generic (PLEG): container finished" podID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerID="d9d7443bf4dafe60fdf0361760f2939824d68daff5a77487250ba830801f2c15" exitCode=7
Jul 14 01:17:27.928158 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:27.927828    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" event={"ID":"ef8ae131-0e3b-4bd5-b19a-ee23801580e2","Type":"ContainerDied","Data":"d9d7443bf4dafe60fdf0361760f2939824d68daff5a77487250ba830801f2c15"}
Jul 14 01:17:27.928158 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:27.927888    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" event={"ID":"ef8ae131-0e3b-4bd5-b19a-ee23801580e2","Type":"ContainerStarted","Data":"0d82c949438936c7194331ffbc5468d0b3aa464dfcc2d47737a41271c9a76571"}
Jul 14 01:17:27.928158 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:27.928047    2552 scope.go:117] "RemoveContainer" containerID="d9d7443bf4dafe60fdf0361760f2939824d68daff5a77487250ba830801f2c15"
Jul 14 01:17:28.933900 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:28.933870    2552 generic.go:356] "Generic (PLEG): container finished" podID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerID="21d8f4f7f1ac5f17c8749dc7db882e908c080ab423b40e05c82a7d76b4c95c36" exitCode=7
Jul 14 01:17:28.934353 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:28.933960    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" event={"ID":"ef8ae131-0e3b-4bd5-b19a-ee23801580e2","Type":"ContainerDied","Data":"21d8f4f7f1ac5f17c8749dc7db882e908c080ab423b40e05c82a7d76b4c95c36"}
Jul 14 01:17:28.934353 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:28.934018    2552 scope.go:117] "RemoveContainer" containerID="d9d7443bf4dafe60fdf0361760f2939824d68daff5a77487250ba830801f2c15"
Jul 14 01:17:28.934353 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:28.934286    2552 scope.go:117] "RemoveContainer" containerID="21d8f4f7f1ac5f17c8749dc7db882e908c080ab423b40e05c82a7d76b4c95c36"
Jul 14 01:17:28.934476 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:17:28.934463    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b3a22390-ftklv_odh-ai-gateway-infra(ef8ae131-0e3b-4bd5-b19a-ee23801580e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2"
Jul 14 01:17:29.939976 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:29.939952    2552 scope.go:117] "RemoveContainer" containerID="21d8f4f7f1ac5f17c8749dc7db882e908c080ab423b40e05c82a7d76b4c95c36"
Jul 14 01:17:29.940344 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:17:29.940132    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"revoke-keys\" with CrashLoopBackOff: \"back-off 10s restarting failed container=revoke-keys pod=maas-api-revoke-keys-e2e-derive-b3a22390-ftklv_odh-ai-gateway-infra(ef8ae131-0e3b-4bd5-b19a-ee23801580e2)\"" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2"
Jul 14 01:17:30.922596 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:30.922568    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:17:40.521000 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:40.520966    2552 scope.go:117] "RemoveContainer" containerID="21d8f4f7f1ac5f17c8749dc7db882e908c080ab423b40e05c82a7d76b4c95c36"
Jul 14 01:17:40.991690 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:40.991656    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" event={"ID":"ef8ae131-0e3b-4bd5-b19a-ee23801580e2","Type":"ContainerStarted","Data":"cbca6774cd22ee6d7eb43b0c2ae08a31169c1f64a9de2a5d805cac83d4591352"}
Jul 14 01:17:41.009119 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:41.009074    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" podStartSLOduration=15.009060257 podStartE2EDuration="15.009060257s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:17:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:17:41.007912549 +0000 UTC m=+2010.988997211" watchObservedRunningTime="2026-07-14 01:17:41.009060257 +0000 UTC m=+2010.990144919"
Jul 14 01:17:41.998322 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:41.998287    2552 generic.go:356] "Generic (PLEG): container finished" podID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerID="cbca6774cd22ee6d7eb43b0c2ae08a31169c1f64a9de2a5d805cac83d4591352" exitCode=0
Jul 14 01:17:41.998322 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:41.998324    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" event={"ID":"ef8ae131-0e3b-4bd5-b19a-ee23801580e2","Type":"ContainerDied","Data":"cbca6774cd22ee6d7eb43b0c2ae08a31169c1f64a9de2a5d805cac83d4591352"}
Jul 14 01:17:41.998825 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:41.998392    2552 scope.go:117] "RemoveContainer" containerID="21d8f4f7f1ac5f17c8749dc7db882e908c080ab423b40e05c82a7d76b4c95c36"
Jul 14 01:17:42.018922 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:42.018897    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"]
Jul 14 01:17:43.138378 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:43.138355    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"
Jul 14 01:17:43.223894 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:43.223860    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae131-0e3b-4bd5-b19a-ee23801580e2-maas-api-service-ca\") pod \"ef8ae131-0e3b-4bd5-b19a-ee23801580e2\" (UID: \"ef8ae131-0e3b-4bd5-b19a-ee23801580e2\") "
Jul 14 01:17:43.224184 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:43.224162    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef8ae131-0e3b-4bd5-b19a-ee23801580e2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ef8ae131-0e3b-4bd5-b19a-ee23801580e2" (UID: "ef8ae131-0e3b-4bd5-b19a-ee23801580e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:17:43.325011 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:43.324990    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef8ae131-0e3b-4bd5-b19a-ee23801580e2-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:17:44.011810 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:44.011778    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"
Jul 14 01:17:44.011987 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:44.011828    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv" event={"ID":"ef8ae131-0e3b-4bd5-b19a-ee23801580e2","Type":"ContainerDied","Data":"0d82c949438936c7194331ffbc5468d0b3aa464dfcc2d47737a41271c9a76571"}
Jul 14 01:17:44.011987 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:44.011884    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d82c949438936c7194331ffbc5468d0b3aa464dfcc2d47737a41271c9a76571"
Jul 14 01:17:44.045120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:44.045098    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"]
Jul 14 01:17:44.048786 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:44.048764    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-ftklv"]
Jul 14 01:17:44.524813 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:17:44.524782    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" path="/var/lib/kubelet/pods/ef8ae131-0e3b-4bd5-b19a-ee23801580e2/volumes"
Jul 14 01:18:16.871018 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.870990    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"]
Jul 14 01:18:16.871487 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.871299    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:18:16.871487 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.871309    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:18:16.871487 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.871323    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:18:16.871487 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.871328    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:18:16.871487 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.871384    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:18:16.871487 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.871394    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:18:16.874192 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.874178    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"
Jul 14 01:18:16.876528 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.876510    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:18:16.880126 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.880097    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"]
Jul 14 01:18:16.905728 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:16.905696    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq\" (UID: \"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"
Jul 14 01:18:17.006315 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:17.006291    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq\" (UID: \"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"
Jul 14 01:18:17.006872 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:17.006854    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq\" (UID: \"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"
Jul 14 01:18:17.185002 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:17.184932    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"
Jul 14 01:18:17.310393 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:17.310365    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"]
Jul 14 01:18:17.313558 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:18:17.313530    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68ed5ba4_b187_46ce_bda5_4bcd57b5cf6d.slice/crio-48451294a105e007b11c7c121ccc6af2b586b4405a7f2beafa63f5a40297f57b WatchSource:0}: Error finding container 48451294a105e007b11c7c121ccc6af2b586b4405a7f2beafa63f5a40297f57b: Status 404 returned error can't find the container with id 48451294a105e007b11c7c121ccc6af2b586b4405a7f2beafa63f5a40297f57b
Jul 14 01:18:18.172259 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:18.172231    2552 generic.go:356] "Generic (PLEG): container finished" podID="68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d" containerID="3a38183877a901a5a0647b51171c0e0eab5ac064c7173341f93fda178cf97da4" exitCode=0
Jul 14 01:18:18.172259 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:18.172267    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq" event={"ID":"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d","Type":"ContainerDied","Data":"3a38183877a901a5a0647b51171c0e0eab5ac064c7173341f93fda178cf97da4"}
Jul 14 01:18:18.172647 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:18.172304    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq" event={"ID":"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d","Type":"ContainerStarted","Data":"48451294a105e007b11c7c121ccc6af2b586b4405a7f2beafa63f5a40297f57b"}
Jul 14 01:18:19.302941 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:19.302917    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"
Jul 14 01:18:19.426902 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:19.426807    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d-maas-api-service-ca\") pod \"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d\" (UID: \"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d\") "
Jul 14 01:18:19.427162 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:19.427141    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d" (UID: "68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:18:19.528095 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:19.528069    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:18:20.182989 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:20.182957    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq" event={"ID":"68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d","Type":"ContainerDied","Data":"48451294a105e007b11c7c121ccc6af2b586b4405a7f2beafa63f5a40297f57b"}
Jul 14 01:18:20.182989 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:20.182981    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq"
Jul 14 01:18:20.183194 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:20.183004    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48451294a105e007b11c7c121ccc6af2b586b4405a7f2beafa63f5a40297f57b"
Jul 14 01:18:27.402990 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.402956    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"]
Jul 14 01:18:27.403370 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.403193    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf" podUID="751fa4b5-b0ce-45ab-b20e-0c12bd56e641" containerName="maas-api" containerID="cri-o://d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307" gracePeriod=30
Jul 14 01:18:27.639478 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.639452    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:18:27.690172 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.690117    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s97f5\" (UniqueName: \"kubernetes.io/projected/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-kube-api-access-s97f5\") pod \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") "
Jul 14 01:18:27.690172 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.690157    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls\") pod \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") "
Jul 14 01:18:27.690354 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.690219    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-openshift-service-ca\") pod \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\" (UID: \"751fa4b5-b0ce-45ab-b20e-0c12bd56e641\") "
Jul 14 01:18:27.690611 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.690589    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "751fa4b5-b0ce-45ab-b20e-0c12bd56e641" (UID: "751fa4b5-b0ce-45ab-b20e-0c12bd56e641"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:18:27.692224 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.692191    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-kube-api-access-s97f5" (OuterVolumeSpecName: "kube-api-access-s97f5") pod "751fa4b5-b0ce-45ab-b20e-0c12bd56e641" (UID: "751fa4b5-b0ce-45ab-b20e-0c12bd56e641"). InnerVolumeSpecName "kube-api-access-s97f5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:18:27.692224 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.692198    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "751fa4b5-b0ce-45ab-b20e-0c12bd56e641" (UID: "751fa4b5-b0ce-45ab-b20e-0c12bd56e641"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:18:27.791385 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.791362    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:18:27.791385 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.791384    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s97f5\" (UniqueName: \"kubernetes.io/projected/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-kube-api-access-s97f5\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:18:27.791527 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:27.791394    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/751fa4b5-b0ce-45ab-b20e-0c12bd56e641-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:18:28.223059 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.223024    2552 generic.go:356] "Generic (PLEG): container finished" podID="751fa4b5-b0ce-45ab-b20e-0c12bd56e641" containerID="d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307" exitCode=0
Jul 14 01:18:28.223195 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.223073    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"
Jul 14 01:18:28.223195 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.223077    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf" event={"ID":"751fa4b5-b0ce-45ab-b20e-0c12bd56e641","Type":"ContainerDied","Data":"d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307"}
Jul 14 01:18:28.223195 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.223185    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf" event={"ID":"751fa4b5-b0ce-45ab-b20e-0c12bd56e641","Type":"ContainerDied","Data":"6f9a81e056dd4b59067f876ccaa82bc5e9fc1e0a8d46d1f67fedca5afbd99478"}
Jul 14 01:18:28.223348 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.223214    2552 scope.go:117] "RemoveContainer" containerID="d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307"
Jul 14 01:18:28.232775 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.232707    2552 scope.go:117] "RemoveContainer" containerID="d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307"
Jul 14 01:18:28.233194 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:18:28.233115    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307\": container with ID starting with d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307 not found: ID does not exist" containerID="d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307"
Jul 14 01:18:28.233194 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.233165    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307"} err="failed to get container status \"d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307\": rpc error: code = NotFound desc = could not find container \"d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307\": container with ID starting with d2e7a5f6ed3ddde8c225eba181540abe5fd25fa52e0e194ca6544d36d6cee307 not found: ID does not exist"
Jul 14 01:18:28.242992 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.242970    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"]
Jul 14 01:18:28.249078 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.249049    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-b3a22390-7b6cbf7b97-dt9kf"]
Jul 14 01:18:28.524814 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:28.524731    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="751fa4b5-b0ce-45ab-b20e-0c12bd56e641" path="/var/lib/kubelet/pods/751fa4b5-b0ce-45ab-b20e-0c12bd56e641/volumes"
Jul 14 01:18:30.230787 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:18:30.230760    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:19:10.709972 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:19:10.709875    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:19:10.716070 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:19:10.715262    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:20:05.267821 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:05.267728    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:20:10.606448 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:10.606421    2552 scope.go:117] "RemoveContainer" containerID="4ffebfa0a4dbfbc105d5b6fc8deb2ac2aba443596bac03cc9a0d136ae0bd322a"
Jul 14 01:20:14.219617 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:14.219578    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:20:20.464782 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.464751    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"]
Jul 14 01:20:20.465122 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465057    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="751fa4b5-b0ce-45ab-b20e-0c12bd56e641" containerName="maas-api"
Jul 14 01:20:20.465122 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465069    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="751fa4b5-b0ce-45ab-b20e-0c12bd56e641" containerName="maas-api"
Jul 14 01:20:20.465122 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465078    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:20:20.465122 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465083    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:20:20.465122 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465096    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d" containerName="revoke-keys"
Jul 14 01:20:20.465122 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465101    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d" containerName="revoke-keys"
Jul 14 01:20:20.465301 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465151    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="751fa4b5-b0ce-45ab-b20e-0c12bd56e641" containerName="maas-api"
Jul 14 01:20:20.465301 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465160    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ef8ae131-0e3b-4bd5-b19a-ee23801580e2" containerName="revoke-keys"
Jul 14 01:20:20.465301 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.465166    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d" containerName="revoke-keys"
Jul 14 01:20:20.468114 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.468089    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.470535 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.470512    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:20:20.471669 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.471650    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:20:20.471780 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.471669    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:20:20.471780 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.471650    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-2a62f7c8-serving-cert\""
Jul 14 01:20:20.478945 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.478921    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"]
Jul 14 01:20:20.544308 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.544284    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-openshift-service-ca\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.544427 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.544317    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-maas-api-tls\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.544427 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.544339    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fl6pp\" (UniqueName: \"kubernetes.io/projected/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-kube-api-access-fl6pp\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.644892 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.644870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-openshift-service-ca\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.645003 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.644909    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-maas-api-tls\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.645003 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.644939    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fl6pp\" (UniqueName: \"kubernetes.io/projected/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-kube-api-access-fl6pp\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.645490 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.645470    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-openshift-service-ca\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.647158 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.647141    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-maas-api-tls\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.652669 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.652642    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fl6pp\" (UniqueName: \"kubernetes.io/projected/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-kube-api-access-fl6pp\") pod \"maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.780600 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.780543    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:20.916553 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.912168    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"]
Jul 14 01:20:20.918003 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:20:20.917976    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9813dcdf_d780_4dd8_a343_ee88c78f1ee2.slice/crio-71c9efbb1b810110f02c39aa039670cf479f76db1abb968f1c2a29d12dfc7797 WatchSource:0}: Error finding container 71c9efbb1b810110f02c39aa039670cf479f76db1abb968f1c2a29d12dfc7797: Status 404 returned error can't find the container with id 71c9efbb1b810110f02c39aa039670cf479f76db1abb968f1c2a29d12dfc7797
Jul 14 01:20:20.919182 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:20.919166    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:20:21.759907 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:21.759824    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp" event={"ID":"9813dcdf-d780-4dd8-a343-ee88c78f1ee2","Type":"ContainerStarted","Data":"89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7"}
Jul 14 01:20:21.759907 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:21.759883    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp" event={"ID":"9813dcdf-d780-4dd8-a343-ee88c78f1ee2","Type":"ContainerStarted","Data":"71c9efbb1b810110f02c39aa039670cf479f76db1abb968f1c2a29d12dfc7797"}
Jul 14 01:20:21.760344 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:21.759925    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:21.775673 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:21.775631    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp" podStartSLOduration=1.204059299 podStartE2EDuration="1.775614833s" totalImagesPullingTime="571.555534ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:20:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:20:21.775003401 +0000 UTC m=+2171.756088054" watchObservedRunningTime="2026-07-14 01:20:21.775614833 +0000 UTC m=+2171.756699497"
Jul 14 01:20:25.428686 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:25.428650    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:20:27.768784 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:27.768758    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:32.031923 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.031885    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"]
Jul 14 01:20:32.035491 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.035475    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"
Jul 14 01:20:32.037835 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.037812    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:20:32.042223 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.042197    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"]
Jul 14 01:20:32.128454 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.128432    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ea6b3a-408b-4965-8bd2-42024ba3217d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc\" (UID: \"a4ea6b3a-408b-4965-8bd2-42024ba3217d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"
Jul 14 01:20:32.229216 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.229193    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ea6b3a-408b-4965-8bd2-42024ba3217d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc\" (UID: \"a4ea6b3a-408b-4965-8bd2-42024ba3217d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"
Jul 14 01:20:32.229763 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.229746    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ea6b3a-408b-4965-8bd2-42024ba3217d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc\" (UID: \"a4ea6b3a-408b-4965-8bd2-42024ba3217d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"
Jul 14 01:20:32.347369 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.347342    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"
Jul 14 01:20:32.479528 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.479483    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"]
Jul 14 01:20:32.481526 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:20:32.481480    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4ea6b3a_408b_4965_8bd2_42024ba3217d.slice/crio-8d2904277b8af0a0e8ee859178eeacdc2ca4fb5fa150eed2e377368b7c0f83ab WatchSource:0}: Error finding container 8d2904277b8af0a0e8ee859178eeacdc2ca4fb5fa150eed2e377368b7c0f83ab: Status 404 returned error can't find the container with id 8d2904277b8af0a0e8ee859178eeacdc2ca4fb5fa150eed2e377368b7c0f83ab
Jul 14 01:20:32.813369 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:32.813341    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc" event={"ID":"a4ea6b3a-408b-4965-8bd2-42024ba3217d","Type":"ContainerStarted","Data":"8d2904277b8af0a0e8ee859178eeacdc2ca4fb5fa150eed2e377368b7c0f83ab"}
Jul 14 01:20:33.819650 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:33.819618    2552 generic.go:356] "Generic (PLEG): container finished" podID="a4ea6b3a-408b-4965-8bd2-42024ba3217d" containerID="ecb27b55d8126d8ff1c94c23bdd174c641be25c6403bcf9010b5cc81086cd412" exitCode=0
Jul 14 01:20:33.820187 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:33.819655    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc" event={"ID":"a4ea6b3a-408b-4965-8bd2-42024ba3217d","Type":"ContainerDied","Data":"ecb27b55d8126d8ff1c94c23bdd174c641be25c6403bcf9010b5cc81086cd412"}
Jul 14 01:20:34.945435 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:34.945415    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"
Jul 14 01:20:35.048140 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:35.048115    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ea6b3a-408b-4965-8bd2-42024ba3217d-maas-api-service-ca\") pod \"a4ea6b3a-408b-4965-8bd2-42024ba3217d\" (UID: \"a4ea6b3a-408b-4965-8bd2-42024ba3217d\") "
Jul 14 01:20:35.048419 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:35.048399    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4ea6b3a-408b-4965-8bd2-42024ba3217d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a4ea6b3a-408b-4965-8bd2-42024ba3217d" (UID: "a4ea6b3a-408b-4965-8bd2-42024ba3217d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:20:35.149085 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:35.149033    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a4ea6b3a-408b-4965-8bd2-42024ba3217d-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:20:35.831827 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:35.831792    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc"
Jul 14 01:20:35.832051 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:35.831815    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc" event={"ID":"a4ea6b3a-408b-4965-8bd2-42024ba3217d","Type":"ContainerDied","Data":"8d2904277b8af0a0e8ee859178eeacdc2ca4fb5fa150eed2e377368b7c0f83ab"}
Jul 14 01:20:35.832051 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:35.831887    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d2904277b8af0a0e8ee859178eeacdc2ca4fb5fa150eed2e377368b7c0f83ab"
Jul 14 01:20:42.578792 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.578752    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"]
Jul 14 01:20:42.579374 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.579009    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp" podUID="9813dcdf-d780-4dd8-a343-ee88c78f1ee2" containerName="maas-api" containerID="cri-o://89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7" gracePeriod=30
Jul 14 01:20:42.822734 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.822713    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:42.867091 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.867025    2552 generic.go:356] "Generic (PLEG): container finished" podID="9813dcdf-d780-4dd8-a343-ee88c78f1ee2" containerID="89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7" exitCode=0
Jul 14 01:20:42.867091 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.867074    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"
Jul 14 01:20:42.867091 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.867074    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp" event={"ID":"9813dcdf-d780-4dd8-a343-ee88c78f1ee2","Type":"ContainerDied","Data":"89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7"}
Jul 14 01:20:42.867277 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.867160    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp" event={"ID":"9813dcdf-d780-4dd8-a343-ee88c78f1ee2","Type":"ContainerDied","Data":"71c9efbb1b810110f02c39aa039670cf479f76db1abb968f1c2a29d12dfc7797"}
Jul 14 01:20:42.867277 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.867190    2552 scope.go:117] "RemoveContainer" containerID="89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7"
Jul 14 01:20:42.875090 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.875073    2552 scope.go:117] "RemoveContainer" containerID="89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7"
Jul 14 01:20:42.875335 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:20:42.875319    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7\": container with ID starting with 89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7 not found: ID does not exist" containerID="89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7"
Jul 14 01:20:42.875385 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.875345    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7"} err="failed to get container status \"89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7\": rpc error: code = NotFound desc = could not find container \"89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7\": container with ID starting with 89d8fd235e1c9dc244faf8f572e2695974b2901b8c8f83f34ffcc871dee72aa7 not found: ID does not exist"
Jul 14 01:20:42.905857 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.905831    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-openshift-service-ca\") pod \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") "
Jul 14 01:20:42.905972 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.905870    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fl6pp\" (UniqueName: \"kubernetes.io/projected/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-kube-api-access-fl6pp\") pod \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") "
Jul 14 01:20:42.905972 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.905903    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-maas-api-tls\") pod \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\" (UID: \"9813dcdf-d780-4dd8-a343-ee88c78f1ee2\") "
Jul 14 01:20:42.906410 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.906378    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "9813dcdf-d780-4dd8-a343-ee88c78f1ee2" (UID: "9813dcdf-d780-4dd8-a343-ee88c78f1ee2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:20:42.908435 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.908407    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "9813dcdf-d780-4dd8-a343-ee88c78f1ee2" (UID: "9813dcdf-d780-4dd8-a343-ee88c78f1ee2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:20:42.908435 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:42.908424    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-kube-api-access-fl6pp" (OuterVolumeSpecName: "kube-api-access-fl6pp") pod "9813dcdf-d780-4dd8-a343-ee88c78f1ee2" (UID: "9813dcdf-d780-4dd8-a343-ee88c78f1ee2"). InnerVolumeSpecName "kube-api-access-fl6pp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:20:43.006751 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:43.006730    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:20:43.006751 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:43.006751    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fl6pp\" (UniqueName: \"kubernetes.io/projected/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-kube-api-access-fl6pp\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:20:43.006872 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:43.006760    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9813dcdf-d780-4dd8-a343-ee88c78f1ee2-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:20:43.188727 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:43.188699    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"]
Jul 14 01:20:43.192057 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:43.192035    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp"]
Jul 14 01:20:44.524873 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:44.524838    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9813dcdf-d780-4dd8-a343-ee88c78f1ee2" path="/var/lib/kubelet/pods/9813dcdf-d780-4dd8-a343-ee88c78f1ee2/volumes"
Jul 14 01:20:45.627171 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:45.627143    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:20:45.765658 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:20:45.765618    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-mt-2a62f7c8-5c4c5c4864-nsvsp" podUID="9813dcdf-d780-4dd8-a343-ee88c78f1ee2" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.37:8443/health\": context deadline exceeded"
Jul 14 01:21:10.624287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:10.624258    2552 scope.go:117] "RemoveContainer" containerID="2f3b6e5c9f6d1096b86f9474ddc6f619bf53f7a22aeebacd4bae0b690a8a761b"
Jul 14 01:21:10.631819 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:10.631793    2552 scope.go:117] "RemoveContainer" containerID="f6a6c52c74760c9bd94ca1aec019d90fc7ba2780640da5cec1b773bba34812d4"
Jul 14 01:21:19.825574 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.825541    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"]
Jul 14 01:21:19.825986 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.825849    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4ea6b3a-408b-4965-8bd2-42024ba3217d" containerName="revoke-keys"
Jul 14 01:21:19.825986 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.825859    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4ea6b3a-408b-4965-8bd2-42024ba3217d" containerName="revoke-keys"
Jul 14 01:21:19.825986 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.825872    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9813dcdf-d780-4dd8-a343-ee88c78f1ee2" containerName="maas-api"
Jul 14 01:21:19.825986 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.825879    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="9813dcdf-d780-4dd8-a343-ee88c78f1ee2" containerName="maas-api"
Jul 14 01:21:19.825986 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.825930    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4ea6b3a-408b-4965-8bd2-42024ba3217d" containerName="revoke-keys"
Jul 14 01:21:19.825986 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.825939    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="9813dcdf-d780-4dd8-a343-ee88c78f1ee2" containerName="maas-api"
Jul 14 01:21:19.828891 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.828872    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:19.831296 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.831274    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-48d1e8ae-serving-cert\""
Jul 14 01:21:19.831424 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.831276    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:21:19.832371 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.832354    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:21:19.832476 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.832354    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:21:19.838569 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.838551    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"]
Jul 14 01:21:19.988764 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.988738    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pph4w\" (UniqueName: \"kubernetes.io/projected/d3931891-9fb8-4b06-9049-0807e2d4cf03-kube-api-access-pph4w\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:19.988892 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.988824    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3931891-9fb8-4b06-9049-0807e2d4cf03-openshift-service-ca\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:19.988892 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:19.988868    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.090113 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.090057    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3931891-9fb8-4b06-9049-0807e2d4cf03-openshift-service-ca\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.090113 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.090097    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.090261 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.090119    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pph4w\" (UniqueName: \"kubernetes.io/projected/d3931891-9fb8-4b06-9049-0807e2d4cf03-kube-api-access-pph4w\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.090261 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:21:20.090227    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-serving-cert: secret "maas-api-e2e-mt-48d1e8ae-serving-cert" not found
Jul 14 01:21:20.090335 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:21:20.090283    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls podName:d3931891-9fb8-4b06-9049-0807e2d4cf03 nodeName:}" failed. No retries permitted until 2026-07-14 01:21:20.590268769 +0000 UTC m=+2230.571353413 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls") pod "maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj" (UID: "d3931891-9fb8-4b06-9049-0807e2d4cf03") : secret "maas-api-e2e-mt-48d1e8ae-serving-cert" not found
Jul 14 01:21:20.090741 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.090725    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3931891-9fb8-4b06-9049-0807e2d4cf03-openshift-service-ca\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.098852 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.098829    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pph4w\" (UniqueName: \"kubernetes.io/projected/d3931891-9fb8-4b06-9049-0807e2d4cf03-kube-api-access-pph4w\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.594372 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.594348    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.596724 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.596693    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls\") pod \"maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.739405 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.739381    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:20.865959 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:20.865940    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"]
Jul 14 01:21:20.868245 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:21:20.868218    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3931891_9fb8_4b06_9049_0807e2d4cf03.slice/crio-e7d9f4b7f69d60fca6663a3e528474d092eabb78fc3e3858b93b304c0514541a WatchSource:0}: Error finding container e7d9f4b7f69d60fca6663a3e528474d092eabb78fc3e3858b93b304c0514541a: Status 404 returned error can't find the container with id e7d9f4b7f69d60fca6663a3e528474d092eabb78fc3e3858b93b304c0514541a
Jul 14 01:21:21.065127 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:21.065091    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj" event={"ID":"d3931891-9fb8-4b06-9049-0807e2d4cf03","Type":"ContainerStarted","Data":"e7d9f4b7f69d60fca6663a3e528474d092eabb78fc3e3858b93b304c0514541a"}
Jul 14 01:21:22.071846 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:22.071812    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj" event={"ID":"d3931891-9fb8-4b06-9049-0807e2d4cf03","Type":"ContainerStarted","Data":"84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760"}
Jul 14 01:21:22.072242 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:22.071933    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:22.096528 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:22.096472    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj" podStartSLOduration=2.695334761 podStartE2EDuration="3.096459787s" totalImagesPullingTime="401.125026ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:21:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:21:22.093562416 +0000 UTC m=+2232.074647079" watchObservedRunningTime="2026-07-14 01:21:22.096459787 +0000 UTC m=+2232.077544448"
Jul 14 01:21:22.235467 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:22.235442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:21:27.823718 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.823688    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"]
Jul 14 01:21:27.827422 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.827405    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:27.830013 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.829990    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-48d1e8ae\"/\"kube-root-ca.crt\""
Jul 14 01:21:27.830013 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.830009    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-48d1e8ae\"/\"default-dockercfg-57hcj\""
Jul 14 01:21:27.830200 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.830009    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-48d1e8ae\"/\"openshift-service-ca.crt\""
Jul 14 01:21:27.831295 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.831273    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-48d1e8ae\"/\"e2e-lifecycle-model-48d1e8ae-kserve-self-signed-certs\""
Jul 14 01:21:27.836206 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.836182    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"]
Jul 14 01:21:27.944862 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.944835    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-home\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:27.944862 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.944863    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-dshm\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:27.945019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.944882    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrbnt\" (UniqueName: \"kubernetes.io/projected/e012be3a-94cc-4138-9ace-e51b912664f2-kube-api-access-jrbnt\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:27.945019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.944913    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012be3a-94cc-4138-9ace-e51b912664f2-tls-certs\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:27.945019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.944957    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-model-cache\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:27.945019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:27.944980    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-tmp-dir\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.045653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.045622    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012be3a-94cc-4138-9ace-e51b912664f2-tls-certs\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.045782 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.045675    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-model-cache\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.045782 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.045707    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-tmp-dir\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.045897 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.045781    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-home\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.045897 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.045804    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-dshm\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.045897 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.045828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrbnt\" (UniqueName: \"kubernetes.io/projected/e012be3a-94cc-4138-9ace-e51b912664f2-kube-api-access-jrbnt\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.046197 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.046173    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-model-cache\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.046283 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.046236    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-home\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.046283 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.046249    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-tmp-dir\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.047990 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.047968    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-dshm\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.048256 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.048236    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012be3a-94cc-4138-9ace-e51b912664f2-tls-certs\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.053339 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.053311    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrbnt\" (UniqueName: \"kubernetes.io/projected/e012be3a-94cc-4138-9ace-e51b912664f2-kube-api-access-jrbnt\") pod \"e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") " pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.081286 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.081243    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:21:28.137035 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.137015    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:28.261431 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:28.261410    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"]
Jul 14 01:21:28.263172 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:21:28.263146    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode012be3a_94cc_4138_9ace_e51b912664f2.slice/crio-a059390ad31a94374937729c944545781e1c6c38391070d840beacfbc21963f2 WatchSource:0}: Error finding container a059390ad31a94374937729c944545781e1c6c38391070d840beacfbc21963f2: Status 404 returned error can't find the container with id a059390ad31a94374937729c944545781e1c6c38391070d840beacfbc21963f2
Jul 14 01:21:29.107044 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:29.107005    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4" event={"ID":"e012be3a-94cc-4138-9ace-e51b912664f2","Type":"ContainerStarted","Data":"49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583"}
Jul 14 01:21:29.107044 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:29.107046    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4" event={"ID":"e012be3a-94cc-4138-9ace-e51b912664f2","Type":"ContainerStarted","Data":"a059390ad31a94374937729c944545781e1c6c38391070d840beacfbc21963f2"}
Jul 14 01:21:29.124027 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:29.123979    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4" podStartSLOduration=2.123964957 podStartE2EDuration="2.123964957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:21:29.121746285 +0000 UTC m=+2239.102830945" watchObservedRunningTime="2026-07-14 01:21:29.123964957 +0000 UTC m=+2239.105049618"
Jul 14 01:21:38.137434 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:38.137341    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:38.137434 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:38.137391    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:38.149734 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:38.149709    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:38.161153 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:38.161132    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:21:39.224030 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:21:39.223997    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:22:09.527168 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:09.527135    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:22:12.871951 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:12.871916    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"]
Jul 14 01:22:12.875638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:12.875623    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"
Jul 14 01:22:12.878133 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:12.878110    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:22:12.880874 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:12.880852    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"]
Jul 14 01:22:12.984357 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:12.984324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be7c25f-9e48-4a28-8974-45005a7080ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb\" (UID: \"6be7c25f-9e48-4a28-8974-45005a7080ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"
Jul 14 01:22:13.085379 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:13.085350    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be7c25f-9e48-4a28-8974-45005a7080ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb\" (UID: \"6be7c25f-9e48-4a28-8974-45005a7080ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"
Jul 14 01:22:13.085944 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:13.085927    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be7c25f-9e48-4a28-8974-45005a7080ca-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb\" (UID: \"6be7c25f-9e48-4a28-8974-45005a7080ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"
Jul 14 01:22:13.187667 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:13.187596    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"
Jul 14 01:22:13.516386 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:13.516346    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"]
Jul 14 01:22:13.518485 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:22:13.518462    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6be7c25f_9e48_4a28_8974_45005a7080ca.slice/crio-08a682d3a6dc957fc751343c2d58ee9a5c7a399690e241cc0d4a706f96725f38 WatchSource:0}: Error finding container 08a682d3a6dc957fc751343c2d58ee9a5c7a399690e241cc0d4a706f96725f38: Status 404 returned error can't find the container with id 08a682d3a6dc957fc751343c2d58ee9a5c7a399690e241cc0d4a706f96725f38
Jul 14 01:22:14.333814 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:14.333772    2552 generic.go:356] "Generic (PLEG): container finished" podID="6be7c25f-9e48-4a28-8974-45005a7080ca" containerID="dd6a29d88e32e105c7c507ae3979f9cd0ded77564be17fdf9aa311a0a1d203f7" exitCode=0
Jul 14 01:22:14.334180 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:14.333827    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb" event={"ID":"6be7c25f-9e48-4a28-8974-45005a7080ca","Type":"ContainerDied","Data":"dd6a29d88e32e105c7c507ae3979f9cd0ded77564be17fdf9aa311a0a1d203f7"}
Jul 14 01:22:14.334180 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:14.333912    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb" event={"ID":"6be7c25f-9e48-4a28-8974-45005a7080ca","Type":"ContainerStarted","Data":"08a682d3a6dc957fc751343c2d58ee9a5c7a399690e241cc0d4a706f96725f38"}
Jul 14 01:22:15.461420 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:15.461397    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"
Jul 14 01:22:15.507974 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:15.507941    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be7c25f-9e48-4a28-8974-45005a7080ca-maas-api-service-ca\") pod \"6be7c25f-9e48-4a28-8974-45005a7080ca\" (UID: \"6be7c25f-9e48-4a28-8974-45005a7080ca\") "
Jul 14 01:22:15.508280 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:15.508259    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6be7c25f-9e48-4a28-8974-45005a7080ca-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6be7c25f-9e48-4a28-8974-45005a7080ca" (UID: "6be7c25f-9e48-4a28-8974-45005a7080ca"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:22:15.608613 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:15.608526    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6be7c25f-9e48-4a28-8974-45005a7080ca-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:16.344723 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:16.344688    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb" event={"ID":"6be7c25f-9e48-4a28-8974-45005a7080ca","Type":"ContainerDied","Data":"08a682d3a6dc957fc751343c2d58ee9a5c7a399690e241cc0d4a706f96725f38"}
Jul 14 01:22:16.344945 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:16.344745    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="08a682d3a6dc957fc751343c2d58ee9a5c7a399690e241cc0d4a706f96725f38"
Jul 14 01:22:16.344945 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:16.344698    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb"
Jul 14 01:22:23.435446 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.435414    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"]
Jul 14 01:22:23.435848 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.435657    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj" podUID="d3931891-9fb8-4b06-9049-0807e2d4cf03" containerName="maas-api" containerID="cri-o://84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760" gracePeriod=30
Jul 14 01:22:23.670970 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.670946    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:22:23.768004 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.767940    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls\") pod \"d3931891-9fb8-4b06-9049-0807e2d4cf03\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") "
Jul 14 01:22:23.768104 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.768006    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pph4w\" (UniqueName: \"kubernetes.io/projected/d3931891-9fb8-4b06-9049-0807e2d4cf03-kube-api-access-pph4w\") pod \"d3931891-9fb8-4b06-9049-0807e2d4cf03\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") "
Jul 14 01:22:23.768104 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.768032    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3931891-9fb8-4b06-9049-0807e2d4cf03-openshift-service-ca\") pod \"d3931891-9fb8-4b06-9049-0807e2d4cf03\" (UID: \"d3931891-9fb8-4b06-9049-0807e2d4cf03\") "
Jul 14 01:22:23.768425 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.768405    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3931891-9fb8-4b06-9049-0807e2d4cf03-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d3931891-9fb8-4b06-9049-0807e2d4cf03" (UID: "d3931891-9fb8-4b06-9049-0807e2d4cf03"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:22:23.770028 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.770003    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d3931891-9fb8-4b06-9049-0807e2d4cf03" (UID: "d3931891-9fb8-4b06-9049-0807e2d4cf03"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:22:23.770118 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.770024    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3931891-9fb8-4b06-9049-0807e2d4cf03-kube-api-access-pph4w" (OuterVolumeSpecName: "kube-api-access-pph4w") pod "d3931891-9fb8-4b06-9049-0807e2d4cf03" (UID: "d3931891-9fb8-4b06-9049-0807e2d4cf03"). InnerVolumeSpecName "kube-api-access-pph4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:22:23.868868 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.868846    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3931891-9fb8-4b06-9049-0807e2d4cf03-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:23.868868 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.868867    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pph4w\" (UniqueName: \"kubernetes.io/projected/d3931891-9fb8-4b06-9049-0807e2d4cf03-kube-api-access-pph4w\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:23.868995 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:23.868878    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3931891-9fb8-4b06-9049-0807e2d4cf03-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:24.386688 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.386652    2552 generic.go:356] "Generic (PLEG): container finished" podID="d3931891-9fb8-4b06-9049-0807e2d4cf03" containerID="84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760" exitCode=0
Jul 14 01:22:24.386843 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.386708    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"
Jul 14 01:22:24.386843 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.386727    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj" event={"ID":"d3931891-9fb8-4b06-9049-0807e2d4cf03","Type":"ContainerDied","Data":"84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760"}
Jul 14 01:22:24.386843 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.386784    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj" event={"ID":"d3931891-9fb8-4b06-9049-0807e2d4cf03","Type":"ContainerDied","Data":"e7d9f4b7f69d60fca6663a3e528474d092eabb78fc3e3858b93b304c0514541a"}
Jul 14 01:22:24.386843 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.386811    2552 scope.go:117] "RemoveContainer" containerID="84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760"
Jul 14 01:22:24.396987 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.396967    2552 scope.go:117] "RemoveContainer" containerID="84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760"
Jul 14 01:22:24.397260 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:22:24.397238    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760\": container with ID starting with 84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760 not found: ID does not exist" containerID="84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760"
Jul 14 01:22:24.397331 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.397269    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760"} err="failed to get container status \"84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760\": rpc error: code = NotFound desc = could not find container \"84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760\": container with ID starting with 84e7292be73de6ea49c754c94e7e4d3232ef11c9dbc1ce745d09bf3101dbf760 not found: ID does not exist"
Jul 14 01:22:24.407880 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.407856    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"]
Jul 14 01:22:24.409703 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.409682    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-48d1e8ae-7cdddc6d97-z98pj"]
Jul 14 01:22:24.524582 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:24.524552    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d3931891-9fb8-4b06-9049-0807e2d4cf03" path="/var/lib/kubelet/pods/d3931891-9fb8-4b06-9049-0807e2d4cf03/volumes"
Jul 14 01:22:26.625723 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:26.625692    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:22:32.943478 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:32.943443    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"]
Jul 14 01:22:47.963047 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:47.962996    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4" podUID="e012be3a-94cc-4138-9ace-e51b912664f2" containerName="main" containerID="cri-o://49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583" gracePeriod=45
Jul 14 01:22:48.209873 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.209852    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:22:48.363012 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.362986    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jrbnt\" (UniqueName: \"kubernetes.io/projected/e012be3a-94cc-4138-9ace-e51b912664f2-kube-api-access-jrbnt\") pod \"e012be3a-94cc-4138-9ace-e51b912664f2\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") "
Jul 14 01:22:48.363154 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363028    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012be3a-94cc-4138-9ace-e51b912664f2-tls-certs\") pod \"e012be3a-94cc-4138-9ace-e51b912664f2\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") "
Jul 14 01:22:48.363154 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363095    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-model-cache\") pod \"e012be3a-94cc-4138-9ace-e51b912664f2\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") "
Jul 14 01:22:48.363154 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363129    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-home\") pod \"e012be3a-94cc-4138-9ace-e51b912664f2\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") "
Jul 14 01:22:48.363312 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363164    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-dshm\") pod \"e012be3a-94cc-4138-9ace-e51b912664f2\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") "
Jul 14 01:22:48.363312 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363209    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-tmp-dir\") pod \"e012be3a-94cc-4138-9ace-e51b912664f2\" (UID: \"e012be3a-94cc-4138-9ace-e51b912664f2\") "
Jul 14 01:22:48.363411 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363387    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-home" (OuterVolumeSpecName: "home") pod "e012be3a-94cc-4138-9ace-e51b912664f2" (UID: "e012be3a-94cc-4138-9ace-e51b912664f2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:22:48.363463 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363425    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-model-cache" (OuterVolumeSpecName: "model-cache") pod "e012be3a-94cc-4138-9ace-e51b912664f2" (UID: "e012be3a-94cc-4138-9ace-e51b912664f2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:22:48.363603 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363570    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e012be3a-94cc-4138-9ace-e51b912664f2" (UID: "e012be3a-94cc-4138-9ace-e51b912664f2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:22:48.363603 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363596    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:48.363776 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.363627    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:48.365213 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.365178    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-dshm" (OuterVolumeSpecName: "dshm") pod "e012be3a-94cc-4138-9ace-e51b912664f2" (UID: "e012be3a-94cc-4138-9ace-e51b912664f2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:22:48.365287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.365244    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e012be3a-94cc-4138-9ace-e51b912664f2-kube-api-access-jrbnt" (OuterVolumeSpecName: "kube-api-access-jrbnt") pod "e012be3a-94cc-4138-9ace-e51b912664f2" (UID: "e012be3a-94cc-4138-9ace-e51b912664f2"). InnerVolumeSpecName "kube-api-access-jrbnt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:22:48.365337 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.365321    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e012be3a-94cc-4138-9ace-e51b912664f2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e012be3a-94cc-4138-9ace-e51b912664f2" (UID: "e012be3a-94cc-4138-9ace-e51b912664f2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:22:48.464909 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.464887    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrbnt\" (UniqueName: \"kubernetes.io/projected/e012be3a-94cc-4138-9ace-e51b912664f2-kube-api-access-jrbnt\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:48.464909 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.464906    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e012be3a-94cc-4138-9ace-e51b912664f2-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:48.465038 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.464914    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:48.465038 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.464922    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e012be3a-94cc-4138-9ace-e51b912664f2-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:22:48.502392 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.502368    2552 generic.go:356] "Generic (PLEG): container finished" podID="e012be3a-94cc-4138-9ace-e51b912664f2" containerID="49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583" exitCode=0
Jul 14 01:22:48.502484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.502420    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"
Jul 14 01:22:48.502484 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.502461    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4" event={"ID":"e012be3a-94cc-4138-9ace-e51b912664f2","Type":"ContainerDied","Data":"49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583"}
Jul 14 01:22:48.502567 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.502518    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4" event={"ID":"e012be3a-94cc-4138-9ace-e51b912664f2","Type":"ContainerDied","Data":"a059390ad31a94374937729c944545781e1c6c38391070d840beacfbc21963f2"}
Jul 14 01:22:48.502567 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.502541    2552 scope.go:117] "RemoveContainer" containerID="49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583"
Jul 14 01:22:48.511327 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.511308    2552 scope.go:117] "RemoveContainer" containerID="49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583"
Jul 14 01:22:48.511661 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:22:48.511642    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583\": container with ID starting with 49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583 not found: ID does not exist" containerID="49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583"
Jul 14 01:22:48.511713 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.511668    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583"} err="failed to get container status \"49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583\": rpc error: code = NotFound desc = could not find container \"49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583\": container with ID starting with 49bb3f133e3b55469228d96a19cbf53eda496999c1b06839a7a4bba992ddf583 not found: ID does not exist"
Jul 14 01:22:48.524409 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.524389    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"]
Jul 14 01:22:48.525464 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:48.525447    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4"]
Jul 14 01:22:50.523843 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:50.523810    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e012be3a-94cc-4138-9ace-e51b912664f2" path="/var/lib/kubelet/pods/e012be3a-94cc-4138-9ace-e51b912664f2/volumes"
Jul 14 01:22:53.151387 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:22:53.151337    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-mt-48d1e8ae/e2e-lifecycle-model-48d1e8ae-kserve-7585ddd64c-bh9s4" podUID="e012be3a-94cc-4138-9ace-e51b912664f2" containerName="main" probeResult="failure" output="Get \"https://10.133.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Jul 14 01:23:10.661102 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:23:10.661039    2552 scope.go:117] "RemoveContainer" containerID="a7c8d4f2456698ba03aaf4aefee3ce687e38e5344046261bb2270ff3e76bbaa2"
Jul 14 01:23:34.235476 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:23:34.235443    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:23:47.523991 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:23:47.523959    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:24:10.685065 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:10.685035    2552 scope.go:117] "RemoveContainer" containerID="cbca6774cd22ee6d7eb43b0c2ae08a31169c1f64a9de2a5d805cac83d4591352"
Jul 14 01:24:10.738110 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:10.738014    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:24:10.745831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:10.745813    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:24:27.923741 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.923711    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"]
Jul 14 01:24:27.924120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924035    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6be7c25f-9e48-4a28-8974-45005a7080ca" containerName="revoke-keys"
Jul 14 01:24:27.924120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924047    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6be7c25f-9e48-4a28-8974-45005a7080ca" containerName="revoke-keys"
Jul 14 01:24:27.924120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924058    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d3931891-9fb8-4b06-9049-0807e2d4cf03" containerName="maas-api"
Jul 14 01:24:27.924120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924064    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d3931891-9fb8-4b06-9049-0807e2d4cf03" containerName="maas-api"
Jul 14 01:24:27.924120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924076    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e012be3a-94cc-4138-9ace-e51b912664f2" containerName="main"
Jul 14 01:24:27.924120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924082    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e012be3a-94cc-4138-9ace-e51b912664f2" containerName="main"
Jul 14 01:24:27.924300 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924134    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e012be3a-94cc-4138-9ace-e51b912664f2" containerName="main"
Jul 14 01:24:27.924300 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924142    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6be7c25f-9e48-4a28-8974-45005a7080ca" containerName="revoke-keys"
Jul 14 01:24:27.924300 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.924150    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d3931891-9fb8-4b06-9049-0807e2d4cf03" containerName="maas-api"
Jul 14 01:24:27.927128 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.927110    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:27.929970 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.929949    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:24:27.930080 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.929969    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:24:27.931069 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.931052    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:24:27.931069 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.931063    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-da3fe6-serving-cert\""
Jul 14 01:24:27.935839 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:27.935419    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"]
Jul 14 01:24:28.102709 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.102683    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-openshift-service-ca\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.102831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.102719    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8hvx\" (UniqueName: \"kubernetes.io/projected/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-kube-api-access-h8hvx\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.102831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.102748    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.203450 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.203390    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-openshift-service-ca\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.203450 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.203429    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8hvx\" (UniqueName: \"kubernetes.io/projected/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-kube-api-access-h8hvx\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.203646 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.203465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.203646 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:24:28.203606    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-serving-cert: secret "maas-api-e2e-inf-a-da3fe6-serving-cert" not found
Jul 14 01:24:28.203743 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:24:28.203673    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls podName:d1c5747f-8ba7-4504-9b3d-8dfeca06e001 nodeName:}" failed. No retries permitted until 2026-07-14 01:24:28.703652073 +0000 UTC m=+2418.684736715 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls") pod "maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" (UID: "d1c5747f-8ba7-4504-9b3d-8dfeca06e001") : secret "maas-api-e2e-inf-a-da3fe6-serving-cert" not found
Jul 14 01:24:28.204142 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.204126    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-openshift-service-ca\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.216426 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.216398    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8hvx\" (UniqueName: \"kubernetes.io/projected/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-kube-api-access-h8hvx\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.707171 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.707136    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.709429 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.709400    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls\") pod \"maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.839645 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.839620    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:28.966141 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:28.966119    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"]
Jul 14 01:24:28.968080 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:24:28.968050    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1c5747f_8ba7_4504_9b3d_8dfeca06e001.slice/crio-34adc1d5a1a9ccb99c1379352d0371dcfeaf7e431eb6c4b0f6a6766697e5b585 WatchSource:0}: Error finding container 34adc1d5a1a9ccb99c1379352d0371dcfeaf7e431eb6c4b0f6a6766697e5b585: Status 404 returned error can't find the container with id 34adc1d5a1a9ccb99c1379352d0371dcfeaf7e431eb6c4b0f6a6766697e5b585
Jul 14 01:24:29.966397 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:29.966360    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" event={"ID":"d1c5747f-8ba7-4504-9b3d-8dfeca06e001","Type":"ContainerStarted","Data":"6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da"}
Jul 14 01:24:29.966831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:29.966414    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:29.966831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:29.966429    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" event={"ID":"d1c5747f-8ba7-4504-9b3d-8dfeca06e001","Type":"ContainerStarted","Data":"34adc1d5a1a9ccb99c1379352d0371dcfeaf7e431eb6c4b0f6a6766697e5b585"}
Jul 14 01:24:29.982787 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:29.982737    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" podStartSLOduration=2.496184499 podStartE2EDuration="2.982721147s" totalImagesPullingTime="486.536648ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:24:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:24:29.981762097 +0000 UTC m=+2419.962846757" watchObservedRunningTime="2026-07-14 01:24:29.982721147 +0000 UTC m=+2419.963805810"
Jul 14 01:24:35.977227 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:35.977141    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:24:36.027372 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.027333    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:24:36.180542 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.180514    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"]
Jul 14 01:24:36.184099 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.184081    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.187439 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.187417    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-cc490a-serving-cert\""
Jul 14 01:24:36.194140 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.194122    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"]
Jul 14 01:24:36.367973 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.367948    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba8a70f-15f8-4f14-90c7-39546b86f663-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.368110 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.367993    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba8a70f-15f8-4f14-90c7-39546b86f663-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.368110 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.368081    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cndc\" (UniqueName: \"kubernetes.io/projected/0ba8a70f-15f8-4f14-90c7-39546b86f663-kube-api-access-9cndc\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.469162 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.469134    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cndc\" (UniqueName: \"kubernetes.io/projected/0ba8a70f-15f8-4f14-90c7-39546b86f663-kube-api-access-9cndc\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.469293 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.469190    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba8a70f-15f8-4f14-90c7-39546b86f663-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.469377 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.469355    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba8a70f-15f8-4f14-90c7-39546b86f663-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.469797 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.469776    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba8a70f-15f8-4f14-90c7-39546b86f663-openshift-service-ca\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.471754 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.471734    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba8a70f-15f8-4f14-90c7-39546b86f663-maas-api-tls\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.478578 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.478551    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cndc\" (UniqueName: \"kubernetes.io/projected/0ba8a70f-15f8-4f14-90c7-39546b86f663-kube-api-access-9cndc\") pod \"maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.496273 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.496254    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:36.621438 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:36.621411    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"]
Jul 14 01:24:36.622745 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:24:36.622717    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ba8a70f_15f8_4f14_90c7_39546b86f663.slice/crio-445ed17573869a1abfb5f7f830ff6860ddea89e79466061ce8a9d67f4c5a1912 WatchSource:0}: Error finding container 445ed17573869a1abfb5f7f830ff6860ddea89e79466061ce8a9d67f4c5a1912: Status 404 returned error can't find the container with id 445ed17573869a1abfb5f7f830ff6860ddea89e79466061ce8a9d67f4c5a1912
Jul 14 01:24:37.000579 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:37.000487    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr" event={"ID":"0ba8a70f-15f8-4f14-90c7-39546b86f663","Type":"ContainerStarted","Data":"445ed17573869a1abfb5f7f830ff6860ddea89e79466061ce8a9d67f4c5a1912"}
Jul 14 01:24:38.007171 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:38.007139    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr" event={"ID":"0ba8a70f-15f8-4f14-90c7-39546b86f663","Type":"ContainerStarted","Data":"03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4"}
Jul 14 01:24:38.007519 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:38.007267    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:38.024751 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:38.024710    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr" podStartSLOduration=1.39316908 podStartE2EDuration="2.02469704s" totalImagesPullingTime="631.52796ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:24:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:24:38.023813986 +0000 UTC m=+2428.004898661" watchObservedRunningTime="2026-07-14 01:24:38.02469704 +0000 UTC m=+2428.005781701"
Jul 14 01:24:44.017708 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:44.017681    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:24:50.026342 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:24:50.026301    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:25:02.729784 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.729753    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"]
Jul 14 01:25:02.733209 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.733191    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.736900 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.736875    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-da3fe6\"/\"kube-root-ca.crt\""
Jul 14 01:25:02.736900 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.736889    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-da3fe6\"/\"default-dockercfg-9cv5j\""
Jul 14 01:25:02.737084 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.736913    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-da3fe6\"/\"openshift-service-ca.crt\""
Jul 14 01:25:02.737084 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.736919    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-da3fe6\"/\"test-model-da3fe6-kserve-self-signed-certs\""
Jul 14 01:25:02.740327 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.740307    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"]
Jul 14 01:25:02.770363 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.770339    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-dshm\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.770468 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.770376    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-home\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.770468 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.770400    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tmp-dir\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.770574 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.770486    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tls-certs\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.770574 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.770526    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-model-cache\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.770643 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.770582    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6rbm\" (UniqueName: \"kubernetes.io/projected/6a6b5500-cf1d-4df9-9a61-2acc1246da35-kube-api-access-r6rbm\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871175 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871148    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-model-cache\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871297 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871194    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6rbm\" (UniqueName: \"kubernetes.io/projected/6a6b5500-cf1d-4df9-9a61-2acc1246da35-kube-api-access-r6rbm\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871297 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871212    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-dshm\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871297 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871248    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-home\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871297 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871270    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tmp-dir\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871598 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871306    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tls-certs\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871598 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871518    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-model-cache\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871708 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871682    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-home\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.871708 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.871695    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tmp-dir\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.873458 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.873438    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-dshm\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.873562 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.873548    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tls-certs\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:02.878255 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:02.878232    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6rbm\" (UniqueName: \"kubernetes.io/projected/6a6b5500-cf1d-4df9-9a61-2acc1246da35-kube-api-access-r6rbm\") pod \"test-model-da3fe6-kserve-799c5cf75f-7q2xs\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") " pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:03.043940 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:03.043868    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:03.167376 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:03.167355    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"]
Jul 14 01:25:03.169466 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:25:03.169442    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6a6b5500_cf1d_4df9_9a61_2acc1246da35.slice/crio-f46fa8182620e321d9b76a333cf86e118ca2f788744d1b1d8c12683d9d4a377a WatchSource:0}: Error finding container f46fa8182620e321d9b76a333cf86e118ca2f788744d1b1d8c12683d9d4a377a: Status 404 returned error can't find the container with id f46fa8182620e321d9b76a333cf86e118ca2f788744d1b1d8c12683d9d4a377a
Jul 14 01:25:04.144184 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:04.144145    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs" event={"ID":"6a6b5500-cf1d-4df9-9a61-2acc1246da35","Type":"ContainerStarted","Data":"2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255"}
Jul 14 01:25:04.144184 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:04.144188    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs" event={"ID":"6a6b5500-cf1d-4df9-9a61-2acc1246da35","Type":"ContainerStarted","Data":"f46fa8182620e321d9b76a333cf86e118ca2f788744d1b1d8c12683d9d4a377a"}
Jul 14 01:25:04.160180 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:04.160132    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs" podStartSLOduration=2.160115412 podStartE2EDuration="2.160115412s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:25:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:25:04.158930456 +0000 UTC m=+2454.140015117" watchObservedRunningTime="2026-07-14 01:25:04.160115412 +0000 UTC m=+2454.141200074"
Jul 14 01:25:05.824019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:05.823988    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:25:13.044287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:13.044253    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:13.044735 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:13.044299    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:13.056465 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:13.056443    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:13.196061 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:13.196042    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:25:17.222543 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:17.222514    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:25:39.128249 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:39.128212    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:25:53.321692 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:53.321662    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:25:54.357742 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:25:54.357712    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"]
Jul 14 01:26:05.722031 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:05.721956    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:26:05.920975 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:05.920945    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"]
Jul 14 01:26:05.921259 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:05.921203    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" podUID="d1c5747f-8ba7-4504-9b3d-8dfeca06e001" containerName="maas-api" containerID="cri-o://6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da" gracePeriod=30
Jul 14 01:26:05.971580 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:05.971549    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" podUID="d1c5747f-8ba7-4504-9b3d-8dfeca06e001" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.42:8443/health\": dial tcp 10.133.0.42:8443: connect: connection refused"
Jul 14 01:26:06.361353 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.361331    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:26:06.397019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.396989    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-openshift-service-ca\") pod \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") "
Jul 14 01:26:06.397162 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.397105    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h8hvx\" (UniqueName: \"kubernetes.io/projected/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-kube-api-access-h8hvx\") pod \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") "
Jul 14 01:26:06.397162 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.397150    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls\") pod \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\" (UID: \"d1c5747f-8ba7-4504-9b3d-8dfeca06e001\") "
Jul 14 01:26:06.397363 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.397340    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d1c5747f-8ba7-4504-9b3d-8dfeca06e001" (UID: "d1c5747f-8ba7-4504-9b3d-8dfeca06e001"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:26:06.397417 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.397406    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:06.399127 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.399103    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d1c5747f-8ba7-4504-9b3d-8dfeca06e001" (UID: "d1c5747f-8ba7-4504-9b3d-8dfeca06e001"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:26:06.399127 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.399106    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-kube-api-access-h8hvx" (OuterVolumeSpecName: "kube-api-access-h8hvx") pod "d1c5747f-8ba7-4504-9b3d-8dfeca06e001" (UID: "d1c5747f-8ba7-4504-9b3d-8dfeca06e001"). InnerVolumeSpecName "kube-api-access-h8hvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:26:06.444886 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.444862    2552 generic.go:356] "Generic (PLEG): container finished" podID="d1c5747f-8ba7-4504-9b3d-8dfeca06e001" containerID="6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da" exitCode=0
Jul 14 01:26:06.444997 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.444907    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"
Jul 14 01:26:06.444997 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.444953    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" event={"ID":"d1c5747f-8ba7-4504-9b3d-8dfeca06e001","Type":"ContainerDied","Data":"6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da"}
Jul 14 01:26:06.445071 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.445005    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt" event={"ID":"d1c5747f-8ba7-4504-9b3d-8dfeca06e001","Type":"ContainerDied","Data":"34adc1d5a1a9ccb99c1379352d0371dcfeaf7e431eb6c4b0f6a6766697e5b585"}
Jul 14 01:26:06.445071 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.445032    2552 scope.go:117] "RemoveContainer" containerID="6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da"
Jul 14 01:26:06.454702 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.453429    2552 scope.go:117] "RemoveContainer" containerID="6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da"
Jul 14 01:26:06.454880 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:26:06.454856    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da\": container with ID starting with 6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da not found: ID does not exist" containerID="6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da"
Jul 14 01:26:06.454937 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.454889    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da"} err="failed to get container status \"6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da\": rpc error: code = NotFound desc = could not find container \"6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da\": container with ID starting with 6b9d9e946e864fc24af244688234fb4927efebe1f76bc4ba3141da2bc23693da not found: ID does not exist"
Jul 14 01:26:06.466036 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.466014    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"]
Jul 14 01:26:06.469236 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.469220    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-da3fe6-7f895d9fc4-fmvbt"]
Jul 14 01:26:06.497893 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.497874    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8hvx\" (UniqueName: \"kubernetes.io/projected/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-kube-api-access-h8hvx\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:06.497965 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.497898    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d1c5747f-8ba7-4504-9b3d-8dfeca06e001-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:06.524411 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:06.524391    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d1c5747f-8ba7-4504-9b3d-8dfeca06e001" path="/var/lib/kubelet/pods/d1c5747f-8ba7-4504-9b3d-8dfeca06e001/volumes"
Jul 14 01:26:09.377616 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.377578    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs" podUID="6a6b5500-cf1d-4df9-9a61-2acc1246da35" containerName="main" containerID="cri-o://2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255" gracePeriod=45
Jul 14 01:26:09.620572 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.620550    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:26:09.721896 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.721768    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-home\") pod \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") "
Jul 14 01:26:09.722055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.721902    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-dshm\") pod \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") "
Jul 14 01:26:09.722055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.721922    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6rbm\" (UniqueName: \"kubernetes.io/projected/6a6b5500-cf1d-4df9-9a61-2acc1246da35-kube-api-access-r6rbm\") pod \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") "
Jul 14 01:26:09.722055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.721978    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-model-cache\") pod \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") "
Jul 14 01:26:09.722055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.721980    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-home" (OuterVolumeSpecName: "home") pod "6a6b5500-cf1d-4df9-9a61-2acc1246da35" (UID: "6a6b5500-cf1d-4df9-9a61-2acc1246da35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:26:09.722055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.722018    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tls-certs\") pod \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") "
Jul 14 01:26:09.722323 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.722066    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tmp-dir\") pod \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\" (UID: \"6a6b5500-cf1d-4df9-9a61-2acc1246da35\") "
Jul 14 01:26:09.722323 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.722215    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-model-cache" (OuterVolumeSpecName: "model-cache") pod "6a6b5500-cf1d-4df9-9a61-2acc1246da35" (UID: "6a6b5500-cf1d-4df9-9a61-2acc1246da35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:26:09.722395 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.722316    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:09.722395 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.722368    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:09.722395 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.722366    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6a6b5500-cf1d-4df9-9a61-2acc1246da35" (UID: "6a6b5500-cf1d-4df9-9a61-2acc1246da35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:26:09.724085 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.724059    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-dshm" (OuterVolumeSpecName: "dshm") pod "6a6b5500-cf1d-4df9-9a61-2acc1246da35" (UID: "6a6b5500-cf1d-4df9-9a61-2acc1246da35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:26:09.724187 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.724091    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6a6b5500-cf1d-4df9-9a61-2acc1246da35" (UID: "6a6b5500-cf1d-4df9-9a61-2acc1246da35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:26:09.724233 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.724183    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6a6b5500-cf1d-4df9-9a61-2acc1246da35-kube-api-access-r6rbm" (OuterVolumeSpecName: "kube-api-access-r6rbm") pod "6a6b5500-cf1d-4df9-9a61-2acc1246da35" (UID: "6a6b5500-cf1d-4df9-9a61-2acc1246da35"). InnerVolumeSpecName "kube-api-access-r6rbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:26:09.829220 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.823982    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:09.829220 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.824025    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6rbm\" (UniqueName: \"kubernetes.io/projected/6a6b5500-cf1d-4df9-9a61-2acc1246da35-kube-api-access-r6rbm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:09.829220 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.824055    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:09.829220 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:09.824070    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6a6b5500-cf1d-4df9-9a61-2acc1246da35-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:10.466456 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.466419    2552 generic.go:356] "Generic (PLEG): container finished" podID="6a6b5500-cf1d-4df9-9a61-2acc1246da35" containerID="2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255" exitCode=0
Jul 14 01:26:10.466890 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.466485    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"
Jul 14 01:26:10.466890 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.466526    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs" event={"ID":"6a6b5500-cf1d-4df9-9a61-2acc1246da35","Type":"ContainerDied","Data":"2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255"}
Jul 14 01:26:10.466890 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.466573    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs" event={"ID":"6a6b5500-cf1d-4df9-9a61-2acc1246da35","Type":"ContainerDied","Data":"f46fa8182620e321d9b76a333cf86e118ca2f788744d1b1d8c12683d9d4a377a"}
Jul 14 01:26:10.466890 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.466591    2552 scope.go:117] "RemoveContainer" containerID="2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255"
Jul 14 01:26:10.475677 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.475657    2552 scope.go:117] "RemoveContainer" containerID="2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255"
Jul 14 01:26:10.475935 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:26:10.475916    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255\": container with ID starting with 2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255 not found: ID does not exist" containerID="2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255"
Jul 14 01:26:10.475988 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.475944    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255"} err="failed to get container status \"2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255\": rpc error: code = NotFound desc = could not find container \"2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255\": container with ID starting with 2f9d59dfff2f15b94e7e40aa6b4b60013bfca78557c33faea7bdf1caeb628255 not found: ID does not exist"
Jul 14 01:26:10.486289 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.486265    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"]
Jul 14 01:26:10.489945 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.489927    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-da3fe6/test-model-da3fe6-kserve-799c5cf75f-7q2xs"]
Jul 14 01:26:10.524733 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:10.524713    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6a6b5500-cf1d-4df9-9a61-2acc1246da35" path="/var/lib/kubelet/pods/6a6b5500-cf1d-4df9-9a61-2acc1246da35/volumes"
Jul 14 01:26:15.626257 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:15.626226    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:26:21.662447 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.662418    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"]
Jul 14 01:26:21.662837 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.662760    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d1c5747f-8ba7-4504-9b3d-8dfeca06e001" containerName="maas-api"
Jul 14 01:26:21.662837 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.662771    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d1c5747f-8ba7-4504-9b3d-8dfeca06e001" containerName="maas-api"
Jul 14 01:26:21.662837 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.662782    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6a6b5500-cf1d-4df9-9a61-2acc1246da35" containerName="main"
Jul 14 01:26:21.662837 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.662787    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6a6b5500-cf1d-4df9-9a61-2acc1246da35" containerName="main"
Jul 14 01:26:21.662969 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.662846    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6a6b5500-cf1d-4df9-9a61-2acc1246da35" containerName="main"
Jul 14 01:26:21.662969 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.662853    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d1c5747f-8ba7-4504-9b3d-8dfeca06e001" containerName="maas-api"
Jul 14 01:26:21.665727 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.665710    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"
Jul 14 01:26:21.668044 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.668023    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:26:21.672230 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.672196    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"]
Jul 14 01:26:21.720924 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.720898    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d5dd0-7825-4d69-b806-80a5c7052bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2\" (UID: \"bb3d5dd0-7825-4d69-b806-80a5c7052bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"
Jul 14 01:26:21.821736 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.821708    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d5dd0-7825-4d69-b806-80a5c7052bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2\" (UID: \"bb3d5dd0-7825-4d69-b806-80a5c7052bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"
Jul 14 01:26:21.822276 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.822251    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d5dd0-7825-4d69-b806-80a5c7052bdd-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2\" (UID: \"bb3d5dd0-7825-4d69-b806-80a5c7052bdd\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"
Jul 14 01:26:21.977038 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:21.976980    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"
Jul 14 01:26:22.107386 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:22.107361    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"]
Jul 14 01:26:22.109903 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:26:22.109874    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb3d5dd0_7825_4d69_b806_80a5c7052bdd.slice/crio-4fd0fd76311868ac6c65135e37063aadf94257199bfe402e0896bc05d6a753c2 WatchSource:0}: Error finding container 4fd0fd76311868ac6c65135e37063aadf94257199bfe402e0896bc05d6a753c2: Status 404 returned error can't find the container with id 4fd0fd76311868ac6c65135e37063aadf94257199bfe402e0896bc05d6a753c2
Jul 14 01:26:22.524055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:22.523974    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2" event={"ID":"bb3d5dd0-7825-4d69-b806-80a5c7052bdd","Type":"ContainerStarted","Data":"34a4b26e79baaaea92e7bcd7e9e92d63964bdd4cd3f3ce4a40e880130f1802da"}
Jul 14 01:26:22.524055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:22.524022    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2" event={"ID":"bb3d5dd0-7825-4d69-b806-80a5c7052bdd","Type":"ContainerStarted","Data":"4fd0fd76311868ac6c65135e37063aadf94257199bfe402e0896bc05d6a753c2"}
Jul 14 01:26:22.540231 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:22.540187    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2" podStartSLOduration=1.5401741549999999 podStartE2EDuration="1.540174155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:26:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:26:22.538251411 +0000 UTC m=+2532.519336073" watchObservedRunningTime="2026-07-14 01:26:22.540174155 +0000 UTC m=+2532.521258815"
Jul 14 01:26:23.530232 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:23.530197    2552 generic.go:356] "Generic (PLEG): container finished" podID="bb3d5dd0-7825-4d69-b806-80a5c7052bdd" containerID="34a4b26e79baaaea92e7bcd7e9e92d63964bdd4cd3f3ce4a40e880130f1802da" exitCode=0
Jul 14 01:26:23.530629 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:23.530293    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2" event={"ID":"bb3d5dd0-7825-4d69-b806-80a5c7052bdd","Type":"ContainerDied","Data":"34a4b26e79baaaea92e7bcd7e9e92d63964bdd4cd3f3ce4a40e880130f1802da"}
Jul 14 01:26:24.657253 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:24.657227    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"
Jul 14 01:26:24.747273 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:24.747242    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d5dd0-7825-4d69-b806-80a5c7052bdd-maas-api-service-ca\") pod \"bb3d5dd0-7825-4d69-b806-80a5c7052bdd\" (UID: \"bb3d5dd0-7825-4d69-b806-80a5c7052bdd\") "
Jul 14 01:26:24.747583 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:24.747561    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb3d5dd0-7825-4d69-b806-80a5c7052bdd-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bb3d5dd0-7825-4d69-b806-80a5c7052bdd" (UID: "bb3d5dd0-7825-4d69-b806-80a5c7052bdd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:26:24.848149 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:24.848129    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bb3d5dd0-7825-4d69-b806-80a5c7052bdd-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:25.540852 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:25.540823    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2"
Jul 14 01:26:25.541016 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:25.540856    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2" event={"ID":"bb3d5dd0-7825-4d69-b806-80a5c7052bdd","Type":"ContainerDied","Data":"4fd0fd76311868ac6c65135e37063aadf94257199bfe402e0896bc05d6a753c2"}
Jul 14 01:26:25.541016 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:25.540898    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4fd0fd76311868ac6c65135e37063aadf94257199bfe402e0896bc05d6a753c2"
Jul 14 01:26:32.050728 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.050693    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"]
Jul 14 01:26:32.051116 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.050895    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr" podUID="0ba8a70f-15f8-4f14-90c7-39546b86f663" containerName="maas-api" containerID="cri-o://03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4" gracePeriod=30
Jul 14 01:26:32.491693 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.491649    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:26:32.571962 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.571930    2552 generic.go:356] "Generic (PLEG): container finished" podID="0ba8a70f-15f8-4f14-90c7-39546b86f663" containerID="03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4" exitCode=0
Jul 14 01:26:32.572118 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.571977    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"
Jul 14 01:26:32.572118 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.572014    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr" event={"ID":"0ba8a70f-15f8-4f14-90c7-39546b86f663","Type":"ContainerDied","Data":"03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4"}
Jul 14 01:26:32.572118 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.572062    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr" event={"ID":"0ba8a70f-15f8-4f14-90c7-39546b86f663","Type":"ContainerDied","Data":"445ed17573869a1abfb5f7f830ff6860ddea89e79466061ce8a9d67f4c5a1912"}
Jul 14 01:26:32.572118 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.572082    2552 scope.go:117] "RemoveContainer" containerID="03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4"
Jul 14 01:26:32.580352 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.580335    2552 scope.go:117] "RemoveContainer" containerID="03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4"
Jul 14 01:26:32.580659 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:26:32.580633    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4\": container with ID starting with 03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4 not found: ID does not exist" containerID="03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4"
Jul 14 01:26:32.580727 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.580667    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4"} err="failed to get container status \"03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4\": rpc error: code = NotFound desc = could not find container \"03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4\": container with ID starting with 03815e6eb8ca215b82d81a90ec00f6c4bb769035a4874fe65f2245ef10fc34c4 not found: ID does not exist"
Jul 14 01:26:32.607925 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.607898    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba8a70f-15f8-4f14-90c7-39546b86f663-maas-api-tls\") pod \"0ba8a70f-15f8-4f14-90c7-39546b86f663\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") "
Jul 14 01:26:32.608008 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.607971    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba8a70f-15f8-4f14-90c7-39546b86f663-openshift-service-ca\") pod \"0ba8a70f-15f8-4f14-90c7-39546b86f663\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") "
Jul 14 01:26:32.608008 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.607988    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9cndc\" (UniqueName: \"kubernetes.io/projected/0ba8a70f-15f8-4f14-90c7-39546b86f663-kube-api-access-9cndc\") pod \"0ba8a70f-15f8-4f14-90c7-39546b86f663\" (UID: \"0ba8a70f-15f8-4f14-90c7-39546b86f663\") "
Jul 14 01:26:32.608284 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.608264    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ba8a70f-15f8-4f14-90c7-39546b86f663-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ba8a70f-15f8-4f14-90c7-39546b86f663" (UID: "0ba8a70f-15f8-4f14-90c7-39546b86f663"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:26:32.609917 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.609892    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ba8a70f-15f8-4f14-90c7-39546b86f663-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ba8a70f-15f8-4f14-90c7-39546b86f663" (UID: "0ba8a70f-15f8-4f14-90c7-39546b86f663"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:26:32.610015 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.609925    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ba8a70f-15f8-4f14-90c7-39546b86f663-kube-api-access-9cndc" (OuterVolumeSpecName: "kube-api-access-9cndc") pod "0ba8a70f-15f8-4f14-90c7-39546b86f663" (UID: "0ba8a70f-15f8-4f14-90c7-39546b86f663"). InnerVolumeSpecName "kube-api-access-9cndc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:26:32.708650 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.708629    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ba8a70f-15f8-4f14-90c7-39546b86f663-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:32.708650 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.708649    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ba8a70f-15f8-4f14-90c7-39546b86f663-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:32.708782 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.708658    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cndc\" (UniqueName: \"kubernetes.io/projected/0ba8a70f-15f8-4f14-90c7-39546b86f663-kube-api-access-9cndc\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:26:32.892895 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.892865    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"]
Jul 14 01:26:32.895277 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:32.895254    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-cc490a-bb975c4b6-htftr"]
Jul 14 01:26:34.524287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:34.524253    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ba8a70f-15f8-4f14-90c7-39546b86f663" path="/var/lib/kubelet/pods/0ba8a70f-15f8-4f14-90c7-39546b86f663/volumes"
Jul 14 01:26:35.227535 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:35.223520    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:26:51.693430 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.693398    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"]
Jul 14 01:26:51.693838 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.693725    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb3d5dd0-7825-4d69-b806-80a5c7052bdd" containerName="revoke-keys"
Jul 14 01:26:51.693838 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.693735    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb3d5dd0-7825-4d69-b806-80a5c7052bdd" containerName="revoke-keys"
Jul 14 01:26:51.693838 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.693745    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ba8a70f-15f8-4f14-90c7-39546b86f663" containerName="maas-api"
Jul 14 01:26:51.693838 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.693751    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ba8a70f-15f8-4f14-90c7-39546b86f663" containerName="maas-api"
Jul 14 01:26:51.693838 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.693802    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ba8a70f-15f8-4f14-90c7-39546b86f663" containerName="maas-api"
Jul 14 01:26:51.693838 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.693814    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb3d5dd0-7825-4d69-b806-80a5c7052bdd" containerName="revoke-keys"
Jul 14 01:26:51.696681 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.696663    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.699093 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.699072    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:26:51.699338 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.699316    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cb7605-serving-cert\""
Jul 14 01:26:51.699441 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.699339    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:26:51.699441 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.699391    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:26:51.705636 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.705614    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"]
Jul 14 01:26:51.862216 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.862188    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bfa46-9650-4e51-879b-2c7e9306203c-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.862372 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.862247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bfa46-9650-4e51-879b-2c7e9306203c-maas-api-tls\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.862372 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.862294    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcjk8\" (UniqueName: \"kubernetes.io/projected/a86bfa46-9650-4e51-879b-2c7e9306203c-kube-api-access-pcjk8\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.963408 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.963342    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bfa46-9650-4e51-879b-2c7e9306203c-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.963408 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.963395    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bfa46-9650-4e51-879b-2c7e9306203c-maas-api-tls\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.963753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.963437    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcjk8\" (UniqueName: \"kubernetes.io/projected/a86bfa46-9650-4e51-879b-2c7e9306203c-kube-api-access-pcjk8\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.963994 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.963977    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bfa46-9650-4e51-879b-2c7e9306203c-openshift-service-ca\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.965720 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.965701    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bfa46-9650-4e51-879b-2c7e9306203c-maas-api-tls\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:51.979034 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:51.979015    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcjk8\" (UniqueName: \"kubernetes.io/projected/a86bfa46-9650-4e51-879b-2c7e9306203c-kube-api-access-pcjk8\") pod \"maas-api-e2e-api-a-cb7605-68dbf6f785-489hm\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:52.007862 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:52.007838    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:52.135874 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:52.135846    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"]
Jul 14 01:26:52.137411 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:26:52.137382    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda86bfa46_9650_4e51_879b_2c7e9306203c.slice/crio-726bed86193050d9dc88ca21a48a99183492bbdd9cb6ea3e580c8ca321a931c6 WatchSource:0}: Error finding container 726bed86193050d9dc88ca21a48a99183492bbdd9cb6ea3e580c8ca321a931c6: Status 404 returned error can't find the container with id 726bed86193050d9dc88ca21a48a99183492bbdd9cb6ea3e580c8ca321a931c6
Jul 14 01:26:52.138801 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:52.138784    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:26:52.665121 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:52.665096    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm" event={"ID":"a86bfa46-9650-4e51-879b-2c7e9306203c","Type":"ContainerStarted","Data":"726bed86193050d9dc88ca21a48a99183492bbdd9cb6ea3e580c8ca321a931c6"}
Jul 14 01:26:53.671179 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:53.671141    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm" event={"ID":"a86bfa46-9650-4e51-879b-2c7e9306203c","Type":"ContainerStarted","Data":"4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647"}
Jul 14 01:26:53.671650 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:53.671232    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:53.688561 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:53.688516    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm" podStartSLOduration=2.185791867 podStartE2EDuration="2.688486082s" totalImagesPullingTime="502.694215ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:26:53.687197778 +0000 UTC m=+2563.668282449" watchObservedRunningTime="2026-07-14 01:26:53.688486082 +0000 UTC m=+2563.669570765"
Jul 14 01:26:57.319153 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:57.319119    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:26:59.683357 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.683328    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"]
Jul 14 01:26:59.687291 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.687267    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:26:59.687439 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.687419    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.689886 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.689864    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-e7c477-serving-cert\""
Jul 14 01:26:59.695533 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.695333    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"]
Jul 14 01:26:59.715145 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.715117    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4739cb48-1077-4e28-8548-609ba45b8811-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.715145 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.715141    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zqgm6\" (UniqueName: \"kubernetes.io/projected/4739cb48-1077-4e28-8548-609ba45b8811-kube-api-access-zqgm6\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.715326 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.715161    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4739cb48-1077-4e28-8548-609ba45b8811-maas-api-tls\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.816249 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.816212    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4739cb48-1077-4e28-8548-609ba45b8811-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.816249 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.816255    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zqgm6\" (UniqueName: \"kubernetes.io/projected/4739cb48-1077-4e28-8548-609ba45b8811-kube-api-access-zqgm6\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.816440 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.816275    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4739cb48-1077-4e28-8548-609ba45b8811-maas-api-tls\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.816868 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.816844    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4739cb48-1077-4e28-8548-609ba45b8811-openshift-service-ca\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.818662 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.818637    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4739cb48-1077-4e28-8548-609ba45b8811-maas-api-tls\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.823981 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.823962    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zqgm6\" (UniqueName: \"kubernetes.io/projected/4739cb48-1077-4e28-8548-609ba45b8811-kube-api-access-zqgm6\") pod \"maas-api-e2e-api-b-e7c477-7569687c84-rbr4h\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:26:59.999736 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:26:59.999664    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:27:00.129102 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:00.129052    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"]
Jul 14 01:27:00.135921 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:27:00.135887    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4739cb48_1077_4e28_8548_609ba45b8811.slice/crio-a85f627e5078958c39cc17ded0a2bd06bfb88065eebbf5b4610afad2a5536e01 WatchSource:0}: Error finding container a85f627e5078958c39cc17ded0a2bd06bfb88065eebbf5b4610afad2a5536e01: Status 404 returned error can't find the container with id a85f627e5078958c39cc17ded0a2bd06bfb88065eebbf5b4610afad2a5536e01
Jul 14 01:27:00.706131 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:00.706098    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h" event={"ID":"4739cb48-1077-4e28-8548-609ba45b8811","Type":"ContainerStarted","Data":"1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3"}
Jul 14 01:27:00.706534 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:00.706152    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h" event={"ID":"4739cb48-1077-4e28-8548-609ba45b8811","Type":"ContainerStarted","Data":"a85f627e5078958c39cc17ded0a2bd06bfb88065eebbf5b4610afad2a5536e01"}
Jul 14 01:27:00.706534 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:00.706193    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:27:00.724930 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:00.724850    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h" podStartSLOduration=1.4036331930000001 podStartE2EDuration="1.724836078s" totalImagesPullingTime="321.202885ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:26:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:27:00.722742336 +0000 UTC m=+2570.703826996" watchObservedRunningTime="2026-07-14 01:27:00.724836078 +0000 UTC m=+2570.705920738"
Jul 14 01:27:06.716583 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:06.716557    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:27:14.136069 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:14.136033    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:27:22.824657 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:22.824630    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"]
Jul 14 01:27:22.825092 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:22.824825    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm" podUID="a86bfa46-9650-4e51-879b-2c7e9306203c" containerName="maas-api" containerID="cri-o://4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647" gracePeriod=30
Jul 14 01:27:23.063209 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.063187    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:27:23.106128 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.106070    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bfa46-9650-4e51-879b-2c7e9306203c-openshift-service-ca\") pod \"a86bfa46-9650-4e51-879b-2c7e9306203c\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") "
Jul 14 01:27:23.106232 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.106166    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pcjk8\" (UniqueName: \"kubernetes.io/projected/a86bfa46-9650-4e51-879b-2c7e9306203c-kube-api-access-pcjk8\") pod \"a86bfa46-9650-4e51-879b-2c7e9306203c\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") "
Jul 14 01:27:23.106232 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.106195    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bfa46-9650-4e51-879b-2c7e9306203c-maas-api-tls\") pod \"a86bfa46-9650-4e51-879b-2c7e9306203c\" (UID: \"a86bfa46-9650-4e51-879b-2c7e9306203c\") "
Jul 14 01:27:23.106438 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.106418    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a86bfa46-9650-4e51-879b-2c7e9306203c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a86bfa46-9650-4e51-879b-2c7e9306203c" (UID: "a86bfa46-9650-4e51-879b-2c7e9306203c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:27:23.108119 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.108091    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a86bfa46-9650-4e51-879b-2c7e9306203c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a86bfa46-9650-4e51-879b-2c7e9306203c" (UID: "a86bfa46-9650-4e51-879b-2c7e9306203c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:27:23.108207 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.108181    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a86bfa46-9650-4e51-879b-2c7e9306203c-kube-api-access-pcjk8" (OuterVolumeSpecName: "kube-api-access-pcjk8") pod "a86bfa46-9650-4e51-879b-2c7e9306203c" (UID: "a86bfa46-9650-4e51-879b-2c7e9306203c"). InnerVolumeSpecName "kube-api-access-pcjk8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:27:23.207742 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.207717    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcjk8\" (UniqueName: \"kubernetes.io/projected/a86bfa46-9650-4e51-879b-2c7e9306203c-kube-api-access-pcjk8\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:27:23.207742 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.207739    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a86bfa46-9650-4e51-879b-2c7e9306203c-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:27:23.207876 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.207748    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a86bfa46-9650-4e51-879b-2c7e9306203c-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:27:23.821948 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.821917    2552 generic.go:356] "Generic (PLEG): container finished" podID="a86bfa46-9650-4e51-879b-2c7e9306203c" containerID="4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647" exitCode=0
Jul 14 01:27:23.822139 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.821960    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"
Jul 14 01:27:23.822139 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.821994    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm" event={"ID":"a86bfa46-9650-4e51-879b-2c7e9306203c","Type":"ContainerDied","Data":"4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647"}
Jul 14 01:27:23.822139 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.822051    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm" event={"ID":"a86bfa46-9650-4e51-879b-2c7e9306203c","Type":"ContainerDied","Data":"726bed86193050d9dc88ca21a48a99183492bbdd9cb6ea3e580c8ca321a931c6"}
Jul 14 01:27:23.822139 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.822079    2552 scope.go:117] "RemoveContainer" containerID="4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647"
Jul 14 01:27:23.830285 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.830082    2552 scope.go:117] "RemoveContainer" containerID="4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647"
Jul 14 01:27:23.830534 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:27:23.830327    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647\": container with ID starting with 4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647 not found: ID does not exist" containerID="4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647"
Jul 14 01:27:23.830534 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.830345    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647"} err="failed to get container status \"4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647\": rpc error: code = NotFound desc = could not find container \"4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647\": container with ID starting with 4c96f8bfb6d082c4a4d3334a301896e44c518793c1e11f52467dd61e32714647 not found: ID does not exist"
Jul 14 01:27:23.840801 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.840771    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"]
Jul 14 01:27:23.844340 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:23.844322    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cb7605-68dbf6f785-489hm"]
Jul 14 01:27:24.524630 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:24.524598    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a86bfa46-9650-4e51-879b-2c7e9306203c" path="/var/lib/kubelet/pods/a86bfa46-9650-4e51-879b-2c7e9306203c/volumes"
Jul 14 01:27:25.328802 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:25.328772    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:27:38.172753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.172679    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"]
Jul 14 01:27:38.173185 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.172994    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a86bfa46-9650-4e51-879b-2c7e9306203c" containerName="maas-api"
Jul 14 01:27:38.173185 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.173006    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a86bfa46-9650-4e51-879b-2c7e9306203c" containerName="maas-api"
Jul 14 01:27:38.173185 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.173082    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a86bfa46-9650-4e51-879b-2c7e9306203c" containerName="maas-api"
Jul 14 01:27:38.177340 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.177323    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"
Jul 14 01:27:38.180168 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.180145    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:27:38.181880 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.181854    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"]
Jul 14 01:27:38.326529 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.326490    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318df0d3-8a01-44bd-a932-a79b75b71323-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z\" (UID: \"318df0d3-8a01-44bd-a932-a79b75b71323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"
Jul 14 01:27:38.427177 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.427122    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318df0d3-8a01-44bd-a932-a79b75b71323-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z\" (UID: \"318df0d3-8a01-44bd-a932-a79b75b71323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"
Jul 14 01:27:38.427749 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.427733    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318df0d3-8a01-44bd-a932-a79b75b71323-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z\" (UID: \"318df0d3-8a01-44bd-a932-a79b75b71323\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"
Jul 14 01:27:38.489127 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.489110    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"
Jul 14 01:27:38.613877 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.613835    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"]
Jul 14 01:27:38.616136 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:27:38.616110    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod318df0d3_8a01_44bd_a932_a79b75b71323.slice/crio-b490e9d359cccef8846ce141c05ad23c6229d0c4d232c882f1d624bb389cf290 WatchSource:0}: Error finding container b490e9d359cccef8846ce141c05ad23c6229d0c4d232c882f1d624bb389cf290: Status 404 returned error can't find the container with id b490e9d359cccef8846ce141c05ad23c6229d0c4d232c882f1d624bb389cf290
Jul 14 01:27:38.892828 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:38.892792    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z" event={"ID":"318df0d3-8a01-44bd-a932-a79b75b71323","Type":"ContainerStarted","Data":"b490e9d359cccef8846ce141c05ad23c6229d0c4d232c882f1d624bb389cf290"}
Jul 14 01:27:39.898445 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:39.898414    2552 generic.go:356] "Generic (PLEG): container finished" podID="318df0d3-8a01-44bd-a932-a79b75b71323" containerID="468e673da14be20ace6692c1c0b9c359413593b47145429bb3c260f60b8cfa6e" exitCode=0
Jul 14 01:27:39.898844 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:39.898514    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z" event={"ID":"318df0d3-8a01-44bd-a932-a79b75b71323","Type":"ContainerDied","Data":"468e673da14be20ace6692c1c0b9c359413593b47145429bb3c260f60b8cfa6e"}
Jul 14 01:27:41.027870 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:41.027847    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"
Jul 14 01:27:41.150710 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:41.150688    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318df0d3-8a01-44bd-a932-a79b75b71323-maas-api-service-ca\") pod \"318df0d3-8a01-44bd-a932-a79b75b71323\" (UID: \"318df0d3-8a01-44bd-a932-a79b75b71323\") "
Jul 14 01:27:41.151011 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:41.150990    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/318df0d3-8a01-44bd-a932-a79b75b71323-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "318df0d3-8a01-44bd-a932-a79b75b71323" (UID: "318df0d3-8a01-44bd-a932-a79b75b71323"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:27:41.252147 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:41.252092    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/318df0d3-8a01-44bd-a932-a79b75b71323-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:27:41.909537 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:41.909486    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z" event={"ID":"318df0d3-8a01-44bd-a932-a79b75b71323","Type":"ContainerDied","Data":"b490e9d359cccef8846ce141c05ad23c6229d0c4d232c882f1d624bb389cf290"}
Jul 14 01:27:41.909537 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:41.909528    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z"
Jul 14 01:27:41.909742 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:41.909556    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b490e9d359cccef8846ce141c05ad23c6229d0c4d232c882f1d624bb389cf290"
Jul 14 01:27:48.613640 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.613609    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"]
Jul 14 01:27:48.614121 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.613807    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h" podUID="4739cb48-1077-4e28-8548-609ba45b8811" containerName="maas-api" containerID="cri-o://1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3" gracePeriod=30
Jul 14 01:27:48.870174 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.870122    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:27:48.945516 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.945468    2552 generic.go:356] "Generic (PLEG): container finished" podID="4739cb48-1077-4e28-8548-609ba45b8811" containerID="1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3" exitCode=0
Jul 14 01:27:48.945653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.945525    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h" event={"ID":"4739cb48-1077-4e28-8548-609ba45b8811","Type":"ContainerDied","Data":"1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3"}
Jul 14 01:27:48.945653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.945564    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h" event={"ID":"4739cb48-1077-4e28-8548-609ba45b8811","Type":"ContainerDied","Data":"a85f627e5078958c39cc17ded0a2bd06bfb88065eebbf5b4610afad2a5536e01"}
Jul 14 01:27:48.945653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.945566    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"
Jul 14 01:27:48.945653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.945583    2552 scope.go:117] "RemoveContainer" containerID="1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3"
Jul 14 01:27:48.953974 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.953959    2552 scope.go:117] "RemoveContainer" containerID="1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3"
Jul 14 01:27:48.954283 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:27:48.954265    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3\": container with ID starting with 1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3 not found: ID does not exist" containerID="1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3"
Jul 14 01:27:48.954329 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:48.954292    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3"} err="failed to get container status \"1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3\": rpc error: code = NotFound desc = could not find container \"1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3\": container with ID starting with 1a5b73667fd7548b1364b152606ea0a34f6412bdcc6f109db4afef7fdcb5e9e3 not found: ID does not exist"
Jul 14 01:27:49.012042 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.012010    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4739cb48-1077-4e28-8548-609ba45b8811-openshift-service-ca\") pod \"4739cb48-1077-4e28-8548-609ba45b8811\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") "
Jul 14 01:27:49.012142 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.012052    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4739cb48-1077-4e28-8548-609ba45b8811-maas-api-tls\") pod \"4739cb48-1077-4e28-8548-609ba45b8811\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") "
Jul 14 01:27:49.012142 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.012102    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zqgm6\" (UniqueName: \"kubernetes.io/projected/4739cb48-1077-4e28-8548-609ba45b8811-kube-api-access-zqgm6\") pod \"4739cb48-1077-4e28-8548-609ba45b8811\" (UID: \"4739cb48-1077-4e28-8548-609ba45b8811\") "
Jul 14 01:27:49.012318 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.012298    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4739cb48-1077-4e28-8548-609ba45b8811-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "4739cb48-1077-4e28-8548-609ba45b8811" (UID: "4739cb48-1077-4e28-8548-609ba45b8811"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:27:49.012418 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.012402    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/4739cb48-1077-4e28-8548-609ba45b8811-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:27:49.014026 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.014005    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4739cb48-1077-4e28-8548-609ba45b8811-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "4739cb48-1077-4e28-8548-609ba45b8811" (UID: "4739cb48-1077-4e28-8548-609ba45b8811"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:27:49.014110 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.014049    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4739cb48-1077-4e28-8548-609ba45b8811-kube-api-access-zqgm6" (OuterVolumeSpecName: "kube-api-access-zqgm6") pod "4739cb48-1077-4e28-8548-609ba45b8811" (UID: "4739cb48-1077-4e28-8548-609ba45b8811"). InnerVolumeSpecName "kube-api-access-zqgm6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:27:49.113114 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.113084    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zqgm6\" (UniqueName: \"kubernetes.io/projected/4739cb48-1077-4e28-8548-609ba45b8811-kube-api-access-zqgm6\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:27:49.113114 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.113115    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/4739cb48-1077-4e28-8548-609ba45b8811-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:27:49.266272 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.266250    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"]
Jul 14 01:27:49.271314 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:49.271293    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-e7c477-7569687c84-rbr4h"]
Jul 14 01:27:50.523914 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:50.523874    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4739cb48-1077-4e28-8548-609ba45b8811" path="/var/lib/kubelet/pods/4739cb48-1077-4e28-8548-609ba45b8811/volumes"
Jul 14 01:27:51.525718 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:27:51.525686    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:28:08.197661 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.197621    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"]
Jul 14 01:28:08.198064 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.198012    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="318df0d3-8a01-44bd-a932-a79b75b71323" containerName="revoke-keys"
Jul 14 01:28:08.198064 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.198024    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="318df0d3-8a01-44bd-a932-a79b75b71323" containerName="revoke-keys"
Jul 14 01:28:08.198064 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.198034    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4739cb48-1077-4e28-8548-609ba45b8811" containerName="maas-api"
Jul 14 01:28:08.198064 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.198042    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4739cb48-1077-4e28-8548-609ba45b8811" containerName="maas-api"
Jul 14 01:28:08.198212 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.198120    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4739cb48-1077-4e28-8548-609ba45b8811" containerName="maas-api"
Jul 14 01:28:08.198212 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.198132    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="318df0d3-8a01-44bd-a932-a79b75b71323" containerName="revoke-keys"
Jul 14 01:28:08.202567 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.202546    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.204923 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.204895    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-99f60e-serving-cert\""
Jul 14 01:28:08.205108 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.205088    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 14 01:28:08.205220 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.205107    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 14 01:28:08.206112 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.206093    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-d6r59\""
Jul 14 01:28:08.210780 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.210761    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"]
Jul 14 01:28:08.371535 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.371485    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d7704d-57cb-44aa-8123-cd2d27bda47e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.371693 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.371617    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw9lr\" (UniqueName: \"kubernetes.io/projected/c0d7704d-57cb-44aa-8123-cd2d27bda47e-kube-api-access-dw9lr\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.371693 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.371683    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d7704d-57cb-44aa-8123-cd2d27bda47e-maas-api-tls\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.472673 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.472600    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw9lr\" (UniqueName: \"kubernetes.io/projected/c0d7704d-57cb-44aa-8123-cd2d27bda47e-kube-api-access-dw9lr\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.472673 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.472644    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d7704d-57cb-44aa-8123-cd2d27bda47e-maas-api-tls\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.472834 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.472676    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d7704d-57cb-44aa-8123-cd2d27bda47e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.473338 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.473321    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d7704d-57cb-44aa-8123-cd2d27bda47e-openshift-service-ca\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.474935 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.474918    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d7704d-57cb-44aa-8123-cd2d27bda47e-maas-api-tls\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.479549 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.479530    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw9lr\" (UniqueName: \"kubernetes.io/projected/c0d7704d-57cb-44aa-8123-cd2d27bda47e-kube-api-access-dw9lr\") pod \"maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.513109 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.513087    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:08.642773 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:08.642750    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"]
Jul 14 01:28:08.644981 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:28:08.644957    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d7704d_57cb_44aa_8123_cd2d27bda47e.slice/crio-a0aff1122f7f201dba315aac3a13537f0969393d3d06c01b61dee31c5ce13f62 WatchSource:0}: Error finding container a0aff1122f7f201dba315aac3a13537f0969393d3d06c01b61dee31c5ce13f62: Status 404 returned error can't find the container with id a0aff1122f7f201dba315aac3a13537f0969393d3d06c01b61dee31c5ce13f62
Jul 14 01:28:09.045108 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:09.045069    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd" event={"ID":"c0d7704d-57cb-44aa-8123-cd2d27bda47e","Type":"ContainerStarted","Data":"a0aff1122f7f201dba315aac3a13537f0969393d3d06c01b61dee31c5ce13f62"}
Jul 14 01:28:10.051701 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:10.051671    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd" event={"ID":"c0d7704d-57cb-44aa-8123-cd2d27bda47e","Type":"ContainerStarted","Data":"4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe"}
Jul 14 01:28:10.052159 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:10.051813    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:10.067222 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:10.067176    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd" podStartSLOduration=1.563598024 podStartE2EDuration="2.067163928s" totalImagesPullingTime="503.565904ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:28:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:28:10.065983302 +0000 UTC m=+2640.047067974" watchObservedRunningTime="2026-07-14 01:28:10.067163928 +0000 UTC m=+2640.048248589"
Jul 14 01:28:13.422820 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:13.422788    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:28:16.063104 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.063078    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:28:16.166477 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.166450    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"]
Jul 14 01:28:16.171030 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.171016    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.173553 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.173529    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-8fd48b-serving-cert\""
Jul 14 01:28:16.178489 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.178371    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"]
Jul 14 01:28:16.230089 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.230062    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.230207 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.230107    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-62s8j\" (UniqueName: \"kubernetes.io/projected/d4ad474c-77ff-40a2-ab33-b752ae1a967d-kube-api-access-62s8j\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.230207 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.230179    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ad474c-77ff-40a2-ab33-b752ae1a967d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.330582 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.330559    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-62s8j\" (UniqueName: \"kubernetes.io/projected/d4ad474c-77ff-40a2-ab33-b752ae1a967d-kube-api-access-62s8j\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.330690 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.330608    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ad474c-77ff-40a2-ab33-b752ae1a967d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.330690 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.330645    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.330761 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:28:16.330734    2552 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-serving-cert: secret "maas-api-e2e-shared-b-8fd48b-serving-cert" not found
Jul 14 01:28:16.330794 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:28:16.330791    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls podName:d4ad474c-77ff-40a2-ab33-b752ae1a967d nodeName:}" failed. No retries permitted until 2026-07-14 01:28:16.830774963 +0000 UTC m=+2646.811859603 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls") pod "maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" (UID: "d4ad474c-77ff-40a2-ab33-b752ae1a967d") : secret "maas-api-e2e-shared-b-8fd48b-serving-cert" not found
Jul 14 01:28:16.331188 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.331170    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ad474c-77ff-40a2-ab33-b752ae1a967d-openshift-service-ca\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.338357 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.338334    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-62s8j\" (UniqueName: \"kubernetes.io/projected/d4ad474c-77ff-40a2-ab33-b752ae1a967d-kube-api-access-62s8j\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.834640 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.834608    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:16.837023 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:16.836996    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls\") pod \"maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:17.081831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:17.081805    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:17.208937 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:17.208909    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"]
Jul 14 01:28:17.210571 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:28:17.210539    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4ad474c_77ff_40a2_ab33_b752ae1a967d.slice/crio-5c8c3ba4aed43f61b366191a16f5e3130cba63c5b27f9e0e3b34e3976564cff4 WatchSource:0}: Error finding container 5c8c3ba4aed43f61b366191a16f5e3130cba63c5b27f9e0e3b34e3976564cff4: Status 404 returned error can't find the container with id 5c8c3ba4aed43f61b366191a16f5e3130cba63c5b27f9e0e3b34e3976564cff4
Jul 14 01:28:18.091820 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:18.091788    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" event={"ID":"d4ad474c-77ff-40a2-ab33-b752ae1a967d","Type":"ContainerStarted","Data":"5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d"}
Jul 14 01:28:18.092207 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:18.091839    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" event={"ID":"d4ad474c-77ff-40a2-ab33-b752ae1a967d","Type":"ContainerStarted","Data":"5c8c3ba4aed43f61b366191a16f5e3130cba63c5b27f9e0e3b34e3976564cff4"}
Jul 14 01:28:18.092207 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:18.091907    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:18.106870 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:18.106829    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" podStartSLOduration=1.72821022 podStartE2EDuration="2.106816374s" totalImagesPullingTime="378.606154ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:28:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:28:18.106112968 +0000 UTC m=+2648.087197627" watchObservedRunningTime="2026-07-14 01:28:18.106816374 +0000 UTC m=+2648.087901118"
Jul 14 01:28:24.101513 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:24.101475    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:28:29.421124 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.421095    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"]
Jul 14 01:28:29.424459 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.424444    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.428114 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.428094    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"kube-root-ca.crt\""
Jul 14 01:28:29.428270 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.428257    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"openshift-service-ca.crt\""
Jul 14 01:28:29.428356 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.428259    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"default-dockercfg-s64g7\""
Jul 14 01:28:29.428356 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.428322    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"auth-test-model-99f60e-kserve-self-signed-certs\""
Jul 14 01:28:29.435005 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.434982    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"]
Jul 14 01:28:29.535300 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.535275    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-dshm\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.535416 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.535313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tls-certs\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.535416 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.535336    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c5x8f\" (UniqueName: \"kubernetes.io/projected/ee5123f7-3e6a-4d7d-9366-361f7546fc79-kube-api-access-c5x8f\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.535558 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.535470    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-home\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.535558 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.535550    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-model-cache\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.535662 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.535578    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tmp-dir\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636296 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636268    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tls-certs\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636392 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c5x8f\" (UniqueName: \"kubernetes.io/projected/ee5123f7-3e6a-4d7d-9366-361f7546fc79-kube-api-access-c5x8f\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636392 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636371    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-home\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636543 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636407    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-model-cache\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636543 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636429    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tmp-dir\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636543 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636468    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-dshm\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636812 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636788    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tmp-dir\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636812 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-model-cache\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.636945 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.636826    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-home\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.638720 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.638692    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tls-certs\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.638720 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.638694    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-dshm\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.643069 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.643051    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c5x8f\" (UniqueName: \"kubernetes.io/projected/ee5123f7-3e6a-4d7d-9366-361f7546fc79-kube-api-access-c5x8f\") pod \"auth-test-model-99f60e-kserve-7df87889ff-bshvm\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") " pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.734051 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.733997    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:29.858677 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.858653    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"]
Jul 14 01:28:29.860149 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:28:29.860125    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee5123f7_3e6a_4d7d_9366_361f7546fc79.slice/crio-0c9179c33f19d772cb440883bb83af4e215546414463ad9288f7b35f43fb1db9 WatchSource:0}: Error finding container 0c9179c33f19d772cb440883bb83af4e215546414463ad9288f7b35f43fb1db9: Status 404 returned error can't find the container with id 0c9179c33f19d772cb440883bb83af4e215546414463ad9288f7b35f43fb1db9
Jul 14 01:28:29.921049 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:29.921028    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:28:30.150256 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:30.150219    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm" event={"ID":"ee5123f7-3e6a-4d7d-9366-361f7546fc79","Type":"ContainerStarted","Data":"3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97"}
Jul 14 01:28:30.150429 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:30.150265    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm" event={"ID":"ee5123f7-3e6a-4d7d-9366-361f7546fc79","Type":"ContainerStarted","Data":"0c9179c33f19d772cb440883bb83af4e215546414463ad9288f7b35f43fb1db9"}
Jul 14 01:28:30.165019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:30.164972    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm" podStartSLOduration=1.164957691 podStartE2EDuration="1.164957691s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:28:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:28:30.164429798 +0000 UTC m=+2660.145514484" watchObservedRunningTime="2026-07-14 01:28:30.164957691 +0000 UTC m=+2660.146042350"
Jul 14 01:28:39.734958 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:39.734923    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:39.735380 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:39.734970    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:39.747311 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:39.747287    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:40.206153 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:40.206125    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:28:41.223434 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:28:41.223403    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:29:04.027064 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.026983    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"]
Jul 14 01:29:04.030862 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.030842    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.033542 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.033520    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"kube-root-ca.crt\""
Jul 14 01:29:04.033635 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.033520    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"openshift-service-ca.crt\""
Jul 14 01:29:04.034984 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.034965    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"default-dockercfg-fwpjf\""
Jul 14 01:29:04.035038 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.034998    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"auth-test-model-8fd48b-kserve-self-signed-certs\""
Jul 14 01:29:04.041861 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.041843    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"]
Jul 14 01:29:04.113452 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.113426    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-model-cache\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.113573 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.113461    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qvh9\" (UniqueName: \"kubernetes.io/projected/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-kube-api-access-5qvh9\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.113573 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.113482    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tmp-dir\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.113573 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.113532    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tls-certs\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.113573 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.113558    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-home\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.113710 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.113593    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-dshm\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.214791 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.214758    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-dshm\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.214901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.214817    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-model-cache\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.214901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.214837    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qvh9\" (UniqueName: \"kubernetes.io/projected/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-kube-api-access-5qvh9\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.214901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.214855    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tmp-dir\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.214901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.214876    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tls-certs\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.214901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.214898    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-home\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.215276 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.215254    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-model-cache\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.215351 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.215263    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tmp-dir\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.215351 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.215301    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-home\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.216933 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.216917    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-dshm\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.217189 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.217172    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tls-certs\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.222210 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.222191    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qvh9\" (UniqueName: \"kubernetes.io/projected/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-kube-api-access-5qvh9\") pod \"auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") " pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.340562 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.340539    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:04.469296 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:04.469265    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"]
Jul 14 01:29:04.470250 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:29:04.470222    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbc77d2b0_b0df_4d93_aaeb_75e30b56ed30.slice/crio-f48c098bcb1c356185ba8c0533b7f75625bf9f71273a7b03678f1f93651d38c6 WatchSource:0}: Error finding container f48c098bcb1c356185ba8c0533b7f75625bf9f71273a7b03678f1f93651d38c6: Status 404 returned error can't find the container with id f48c098bcb1c356185ba8c0533b7f75625bf9f71273a7b03678f1f93651d38c6
Jul 14 01:29:05.317367 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:05.317328    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc" event={"ID":"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30","Type":"ContainerStarted","Data":"4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb"}
Jul 14 01:29:05.317367 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:05.317371    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc" event={"ID":"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30","Type":"ContainerStarted","Data":"f48c098bcb1c356185ba8c0533b7f75625bf9f71273a7b03678f1f93651d38c6"}
Jul 14 01:29:05.333663 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:05.333623    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc" podStartSLOduration=1.333608624 podStartE2EDuration="1.333608624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:29:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:29:05.332908671 +0000 UTC m=+2695.313993331" watchObservedRunningTime="2026-07-14 01:29:05.333608624 +0000 UTC m=+2695.314693281"
Jul 14 01:29:08.123283 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:08.123249    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:29:10.773692 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:10.773589    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:29:10.788899 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:10.782222    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:29:14.340942 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:14.340911    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:14.341373 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:14.340960    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:14.353121 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:14.353102    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:14.372262 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:14.372234    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:29:22.524762 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:22.524732    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:29:41.131104 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:41.131048    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:29:57.021792 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:29:57.021760    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:30:14.322880 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:30:14.322844    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:30:31.331152 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:30:31.331122    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:30:48.820552 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:30:48.820452    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:31:02.639562 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:02.639527    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"]
Jul 14 01:31:03.605561 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:03.605527    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"]
Jul 14 01:31:07.123766 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:07.123736    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:31:17.897400 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:17.897359    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm" podUID="ee5123f7-3e6a-4d7d-9366-361f7546fc79" containerName="main" containerID="cri-o://3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97" gracePeriod=45
Jul 14 01:31:18.144836 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.144814    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:31:18.307177 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307095    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tls-certs\") pod \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") "
Jul 14 01:31:18.307177 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307143    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tmp-dir\") pod \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") "
Jul 14 01:31:18.307177 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307171    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-model-cache\") pod \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") "
Jul 14 01:31:18.307447 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307211    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-dshm\") pod \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") "
Jul 14 01:31:18.307447 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307286    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c5x8f\" (UniqueName: \"kubernetes.io/projected/ee5123f7-3e6a-4d7d-9366-361f7546fc79-kube-api-access-c5x8f\") pod \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") "
Jul 14 01:31:18.307447 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307325    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-home\") pod \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\" (UID: \"ee5123f7-3e6a-4d7d-9366-361f7546fc79\") "
Jul 14 01:31:18.307447 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307402    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ee5123f7-3e6a-4d7d-9366-361f7546fc79" (UID: "ee5123f7-3e6a-4d7d-9366-361f7546fc79"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:18.307682 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307410    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-model-cache" (OuterVolumeSpecName: "model-cache") pod "ee5123f7-3e6a-4d7d-9366-361f7546fc79" (UID: "ee5123f7-3e6a-4d7d-9366-361f7546fc79"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:18.307682 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307624    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-home" (OuterVolumeSpecName: "home") pod "ee5123f7-3e6a-4d7d-9366-361f7546fc79" (UID: "ee5123f7-3e6a-4d7d-9366-361f7546fc79"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:18.307682 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307673    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:18.307812 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.307691    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:18.309288 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.309268    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ee5123f7-3e6a-4d7d-9366-361f7546fc79" (UID: "ee5123f7-3e6a-4d7d-9366-361f7546fc79"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:31:18.309720 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.309701    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-dshm" (OuterVolumeSpecName: "dshm") pod "ee5123f7-3e6a-4d7d-9366-361f7546fc79" (UID: "ee5123f7-3e6a-4d7d-9366-361f7546fc79"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:18.309720 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.309709    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee5123f7-3e6a-4d7d-9366-361f7546fc79-kube-api-access-c5x8f" (OuterVolumeSpecName: "kube-api-access-c5x8f") pod "ee5123f7-3e6a-4d7d-9366-361f7546fc79" (UID: "ee5123f7-3e6a-4d7d-9366-361f7546fc79"). InnerVolumeSpecName "kube-api-access-c5x8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:31:18.408180 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.408150    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c5x8f\" (UniqueName: \"kubernetes.io/projected/ee5123f7-3e6a-4d7d-9366-361f7546fc79-kube-api-access-c5x8f\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:18.408180 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.408177    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:18.408335 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.408191    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ee5123f7-3e6a-4d7d-9366-361f7546fc79-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:18.408335 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.408204    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ee5123f7-3e6a-4d7d-9366-361f7546fc79-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:18.624983 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.624944    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc" podUID="bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" containerName="main" containerID="cri-o://4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb" gracePeriod=45
Jul 14 01:31:18.857739 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.857716    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:31:18.962022 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.961993    2552 generic.go:356] "Generic (PLEG): container finished" podID="ee5123f7-3e6a-4d7d-9366-361f7546fc79" containerID="3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97" exitCode=0
Jul 14 01:31:18.962423 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.962032    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm" event={"ID":"ee5123f7-3e6a-4d7d-9366-361f7546fc79","Type":"ContainerDied","Data":"3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97"}
Jul 14 01:31:18.962423 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.962057    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"
Jul 14 01:31:18.962423 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.962069    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm" event={"ID":"ee5123f7-3e6a-4d7d-9366-361f7546fc79","Type":"ContainerDied","Data":"0c9179c33f19d772cb440883bb83af4e215546414463ad9288f7b35f43fb1db9"}
Jul 14 01:31:18.962423 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.962084    2552 scope.go:117] "RemoveContainer" containerID="3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97"
Jul 14 01:31:18.963478 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.963441    2552 generic.go:356] "Generic (PLEG): container finished" podID="bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" containerID="4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb" exitCode=0
Jul 14 01:31:18.963573 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.963490    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"
Jul 14 01:31:18.963573 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.963546    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc" event={"ID":"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30","Type":"ContainerDied","Data":"4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb"}
Jul 14 01:31:18.963660 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.963587    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc" event={"ID":"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30","Type":"ContainerDied","Data":"f48c098bcb1c356185ba8c0533b7f75625bf9f71273a7b03678f1f93651d38c6"}
Jul 14 01:31:18.970566 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.970548    2552 scope.go:117] "RemoveContainer" containerID="3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97"
Jul 14 01:31:18.970811 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:31:18.970793    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97\": container with ID starting with 3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97 not found: ID does not exist" containerID="3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97"
Jul 14 01:31:18.970876 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.970817    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97"} err="failed to get container status \"3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97\": rpc error: code = NotFound desc = could not find container \"3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97\": container with ID starting with 3282b6ad01cf600b4db0b6ab8e5ae6f2fa8c8662efe84c9d0c4f57da7fd2df97 not found: ID does not exist"
Jul 14 01:31:18.970876 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.970837    2552 scope.go:117] "RemoveContainer" containerID="4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb"
Jul 14 01:31:18.977787 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.977767    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"]
Jul 14 01:31:18.978176 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.978163    2552 scope.go:117] "RemoveContainer" containerID="4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb"
Jul 14 01:31:18.978418 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:31:18.978402    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb\": container with ID starting with 4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb not found: ID does not exist" containerID="4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb"
Jul 14 01:31:18.978471 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.978422    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb"} err="failed to get container status \"4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb\": rpc error: code = NotFound desc = could not find container \"4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb\": container with ID starting with 4ad8f5c6c2d35aa94a548610ec96ab7602a49f18b0652d3bbe0f2eb061d7ebeb not found: ID does not exist"
Jul 14 01:31:18.981393 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:18.981373    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/auth-test-model-99f60e-kserve-7df87889ff-bshvm"]
Jul 14 01:31:19.014581 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014562    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-home\") pod \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") "
Jul 14 01:31:19.014659 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014599    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tls-certs\") pod \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") "
Jul 14 01:31:19.014659 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014622    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5qvh9\" (UniqueName: \"kubernetes.io/projected/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-kube-api-access-5qvh9\") pod \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") "
Jul 14 01:31:19.014659 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014648    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-dshm\") pod \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") "
Jul 14 01:31:19.014820 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014683    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-model-cache\") pod \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") "
Jul 14 01:31:19.014820 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014737    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tmp-dir\") pod \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\" (UID: \"bc77d2b0-b0df-4d93-aaeb-75e30b56ed30\") "
Jul 14 01:31:19.014914 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014829    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-home" (OuterVolumeSpecName: "home") pod "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" (UID: "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:19.015010 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.014983    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-model-cache" (OuterVolumeSpecName: "model-cache") pod "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" (UID: "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:19.015079 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.015028    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:19.015130 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.015104    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" (UID: "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:19.016689 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.016668    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" (UID: "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:31:19.016752 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.016669    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-dshm" (OuterVolumeSpecName: "dshm") pod "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" (UID: "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:31:19.016752 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.016729    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-kube-api-access-5qvh9" (OuterVolumeSpecName: "kube-api-access-5qvh9") pod "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" (UID: "bc77d2b0-b0df-4d93-aaeb-75e30b56ed30"). InnerVolumeSpecName "kube-api-access-5qvh9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:31:19.115947 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.115924    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:19.115947 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.115944    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:19.116067 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.115952    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:19.116067 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.115961    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5qvh9\" (UniqueName: \"kubernetes.io/projected/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-kube-api-access-5qvh9\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:19.116067 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.115970    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:31:19.285886 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.285862    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"]
Jul 14 01:31:19.289132 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:19.289110    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/auth-test-model-8fd48b-kserve-7cdccc9499-c8qjc"]
Jul 14 01:31:20.524120 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:20.524082    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" path="/var/lib/kubelet/pods/bc77d2b0-b0df-4d93-aaeb-75e30b56ed30/volumes"
Jul 14 01:31:20.524650 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:20.524631    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee5123f7-3e6a-4d7d-9366-361f7546fc79" path="/var/lib/kubelet/pods/ee5123f7-3e6a-4d7d-9366-361f7546fc79/volumes"
Jul 14 01:31:24.129793 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:24.129763    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:31:40.822808 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.822775    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"]
Jul 14 01:31:40.823225 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.823130    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" containerName="main"
Jul 14 01:31:40.823225 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.823141    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" containerName="main"
Jul 14 01:31:40.823225 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.823153    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee5123f7-3e6a-4d7d-9366-361f7546fc79" containerName="main"
Jul 14 01:31:40.823225 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.823159    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee5123f7-3e6a-4d7d-9366-361f7546fc79" containerName="main"
Jul 14 01:31:40.823371 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.823230    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc77d2b0-b0df-4d93-aaeb-75e30b56ed30" containerName="main"
Jul 14 01:31:40.823371 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.823241    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee5123f7-3e6a-4d7d-9366-361f7546fc79" containerName="main"
Jul 14 01:31:40.833059 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.833034    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.835265 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.835228    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"]
Jul 14 01:31:40.836753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.836559    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"default-dockercfg-fwpjf\""
Jul 14 01:31:40.836753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.836595    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"kube-root-ca.crt\""
Jul 14 01:31:40.836753 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.836559    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"openshift-service-ca.crt\""
Jul 14 01:31:40.837062 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.837042    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"sub-test-model-8fd48b-kserve-self-signed-certs\""
Jul 14 01:31:40.875646 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.875621    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-home\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.875760 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.875677    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-model-cache\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.875760 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.875694    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-tmp-dir\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.875760 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.875723    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm6fd\" (UniqueName: \"kubernetes.io/projected/2171b443-cac7-48c3-ab04-3e6140ac8d30-kube-api-access-wm6fd\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.875760 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.875742    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171b443-cac7-48c3-ab04-3e6140ac8d30-tls-certs\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.875920 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.875759    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-dshm\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977027 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.976996    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-model-cache\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977159 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977032    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-tmp-dir\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977159 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977057    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm6fd\" (UniqueName: \"kubernetes.io/projected/2171b443-cac7-48c3-ab04-3e6140ac8d30-kube-api-access-wm6fd\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977159 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977082    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171b443-cac7-48c3-ab04-3e6140ac8d30-tls-certs\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977159 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977110    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-dshm\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977366 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977170    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-home\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977463 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977431    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-model-cache\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977463 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977434    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-tmp-dir\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.977638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.977486    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-home\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.979313 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.979291    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-dshm\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.979530 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.979514    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171b443-cac7-48c3-ab04-3e6140ac8d30-tls-certs\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:40.984129 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:40.984111    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm6fd\" (UniqueName: \"kubernetes.io/projected/2171b443-cac7-48c3-ab04-3e6140ac8d30-kube-api-access-wm6fd\") pod \"sub-test-model-8fd48b-kserve-6964b649cf-g6j6h\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") " pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:41.143759 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:41.143682    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:41.271173 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:41.271151    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"]
Jul 14 01:31:41.273372 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:31:41.273341    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2171b443_cac7_48c3_ab04_3e6140ac8d30.slice/crio-9484fb5dcb66ad9df8b11563ec63d26acc1952d38bdbdb7ea3dbfc83f60a965b WatchSource:0}: Error finding container 9484fb5dcb66ad9df8b11563ec63d26acc1952d38bdbdb7ea3dbfc83f60a965b: Status 404 returned error can't find the container with id 9484fb5dcb66ad9df8b11563ec63d26acc1952d38bdbdb7ea3dbfc83f60a965b
Jul 14 01:31:42.078057 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:42.078023    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h" event={"ID":"2171b443-cac7-48c3-ab04-3e6140ac8d30","Type":"ContainerStarted","Data":"f341cb08c0a90f80145da8ffc01461c07a9771db42aaaddc8cb86dbc7f8afde2"}
Jul 14 01:31:42.078420 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:42.078073    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h" event={"ID":"2171b443-cac7-48c3-ab04-3e6140ac8d30","Type":"ContainerStarted","Data":"9484fb5dcb66ad9df8b11563ec63d26acc1952d38bdbdb7ea3dbfc83f60a965b"}
Jul 14 01:31:42.093947 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:42.093902    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h" podStartSLOduration=2.093886773 podStartE2EDuration="2.093886773s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:31:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:31:42.091732592 +0000 UTC m=+2852.072817252" watchObservedRunningTime="2026-07-14 01:31:42.093886773 +0000 UTC m=+2852.074971477"
Jul 14 01:31:45.125166 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:45.125130    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:31:51.143847 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:51.143810    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:51.144253 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:51.143863    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:51.156097 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:51.156077    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:31:52.137141 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:31:52.137112    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:32:00.826173 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:00.826141    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:32:18.138262 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:18.138230    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"]
Jul 14 01:32:22.025063 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:22.025025    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:32:33.157825 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.157759    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h" podUID="2171b443-cac7-48c3-ab04-3e6140ac8d30" containerName="main" containerID="cri-o://f341cb08c0a90f80145da8ffc01461c07a9771db42aaaddc8cb86dbc7f8afde2" gracePeriod=45
Jul 14 01:32:33.321837 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.321807    2552 generic.go:356] "Generic (PLEG): container finished" podID="2171b443-cac7-48c3-ab04-3e6140ac8d30" containerID="f341cb08c0a90f80145da8ffc01461c07a9771db42aaaddc8cb86dbc7f8afde2" exitCode=0
Jul 14 01:32:33.321994 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.321877    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h" event={"ID":"2171b443-cac7-48c3-ab04-3e6140ac8d30","Type":"ContainerDied","Data":"f341cb08c0a90f80145da8ffc01461c07a9771db42aaaddc8cb86dbc7f8afde2"}
Jul 14 01:32:33.403924 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.403897    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:32:33.521600 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521485    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-home\") pod \"2171b443-cac7-48c3-ab04-3e6140ac8d30\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") "
Jul 14 01:32:33.521600 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521547    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171b443-cac7-48c3-ab04-3e6140ac8d30-tls-certs\") pod \"2171b443-cac7-48c3-ab04-3e6140ac8d30\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") "
Jul 14 01:32:33.521600 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521573    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-dshm\") pod \"2171b443-cac7-48c3-ab04-3e6140ac8d30\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") "
Jul 14 01:32:33.521901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521622    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wm6fd\" (UniqueName: \"kubernetes.io/projected/2171b443-cac7-48c3-ab04-3e6140ac8d30-kube-api-access-wm6fd\") pod \"2171b443-cac7-48c3-ab04-3e6140ac8d30\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") "
Jul 14 01:32:33.521901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521660    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-model-cache\") pod \"2171b443-cac7-48c3-ab04-3e6140ac8d30\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") "
Jul 14 01:32:33.521901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521682    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-tmp-dir\") pod \"2171b443-cac7-48c3-ab04-3e6140ac8d30\" (UID: \"2171b443-cac7-48c3-ab04-3e6140ac8d30\") "
Jul 14 01:32:33.521901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521772    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-home" (OuterVolumeSpecName: "home") pod "2171b443-cac7-48c3-ab04-3e6140ac8d30" (UID: "2171b443-cac7-48c3-ab04-3e6140ac8d30"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:33.522070 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521952    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:32:33.522070 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521966    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2171b443-cac7-48c3-ab04-3e6140ac8d30" (UID: "2171b443-cac7-48c3-ab04-3e6140ac8d30"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:33.522070 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.521986    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-model-cache" (OuterVolumeSpecName: "model-cache") pod "2171b443-cac7-48c3-ab04-3e6140ac8d30" (UID: "2171b443-cac7-48c3-ab04-3e6140ac8d30"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:33.523720 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.523698    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-dshm" (OuterVolumeSpecName: "dshm") pod "2171b443-cac7-48c3-ab04-3e6140ac8d30" (UID: "2171b443-cac7-48c3-ab04-3e6140ac8d30"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:32:33.523818 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.523798    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2171b443-cac7-48c3-ab04-3e6140ac8d30-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2171b443-cac7-48c3-ab04-3e6140ac8d30" (UID: "2171b443-cac7-48c3-ab04-3e6140ac8d30"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:32:33.523818 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.523807    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2171b443-cac7-48c3-ab04-3e6140ac8d30-kube-api-access-wm6fd" (OuterVolumeSpecName: "kube-api-access-wm6fd") pod "2171b443-cac7-48c3-ab04-3e6140ac8d30" (UID: "2171b443-cac7-48c3-ab04-3e6140ac8d30"). InnerVolumeSpecName "kube-api-access-wm6fd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:32:33.622442 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.622401    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:32:33.622442 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.622432    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wm6fd\" (UniqueName: \"kubernetes.io/projected/2171b443-cac7-48c3-ab04-3e6140ac8d30-kube-api-access-wm6fd\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:32:33.622442 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.622447    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:32:33.622707 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.622474    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2171b443-cac7-48c3-ab04-3e6140ac8d30-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:32:33.622707 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:33.622482    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2171b443-cac7-48c3-ab04-3e6140ac8d30-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:32:34.327831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:34.327796    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h" event={"ID":"2171b443-cac7-48c3-ab04-3e6140ac8d30","Type":"ContainerDied","Data":"9484fb5dcb66ad9df8b11563ec63d26acc1952d38bdbdb7ea3dbfc83f60a965b"}
Jul 14 01:32:34.327831 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:34.327829    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"
Jul 14 01:32:34.328314 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:34.327848    2552 scope.go:117] "RemoveContainer" containerID="f341cb08c0a90f80145da8ffc01461c07a9771db42aaaddc8cb86dbc7f8afde2"
Jul 14 01:32:34.347929 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:34.347894    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"]
Jul 14 01:32:34.351411 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:34.351387    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/sub-test-model-8fd48b-kserve-6964b649cf-g6j6h"]
Jul 14 01:32:34.524329 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:34.524300    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2171b443-cac7-48c3-ab04-3e6140ac8d30" path="/var/lib/kubelet/pods/2171b443-cac7-48c3-ab04-3e6140ac8d30/volumes"
Jul 14 01:32:39.413725 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.413694    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"]
Jul 14 01:32:39.414235 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.414215    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2171b443-cac7-48c3-ab04-3e6140ac8d30" containerName="main"
Jul 14 01:32:39.414323 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.414238    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2171b443-cac7-48c3-ab04-3e6140ac8d30" containerName="main"
Jul 14 01:32:39.414375 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.414334    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2171b443-cac7-48c3-ab04-3e6140ac8d30" containerName="main"
Jul 14 01:32:39.418904 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.418882    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.422763 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.422738    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"openshift-service-ca.crt\""
Jul 14 01:32:39.422887 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.422739    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"default-dockercfg-s64g7\""
Jul 14 01:32:39.422887 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.422747    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"kube-root-ca.crt\""
Jul 14 01:32:39.422887 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.422846    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-99f60e\"/\"rate-test-model-99f60e-kserve-self-signed-certs\""
Jul 14 01:32:39.425758 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.425526    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"]
Jul 14 01:32:39.470633 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.470595    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-dshm\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.470633 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.470636    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e931986-a1d8-4aee-8df0-3908ef964f5e-tls-certs\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.470858 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.470665    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-home\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.470858 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.470746    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-model-cache\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.470858 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.470788    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-tmp-dir\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.470858 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.470807    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fft4r\" (UniqueName: \"kubernetes.io/projected/7e931986-a1d8-4aee-8df0-3908ef964f5e-kube-api-access-fft4r\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571368 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571330    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-tmp-dir\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571368 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571369    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fft4r\" (UniqueName: \"kubernetes.io/projected/7e931986-a1d8-4aee-8df0-3908ef964f5e-kube-api-access-fft4r\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571624 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571411    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-dshm\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571624 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571443    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e931986-a1d8-4aee-8df0-3908ef964f5e-tls-certs\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571624 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571478    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-home\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571624 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571548    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-model-cache\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571820 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571800    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-tmp-dir\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571873 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571854    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-model-cache\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.571918 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.571894    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-home\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.573722 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.573700    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-dshm\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.573887 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.573870    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e931986-a1d8-4aee-8df0-3908ef964f5e-tls-certs\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.579557 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.579527    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fft4r\" (UniqueName: \"kubernetes.io/projected/7e931986-a1d8-4aee-8df0-3908ef964f5e-kube-api-access-fft4r\") pod \"rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") " pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.729174 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.729084    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:39.855864 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:39.855818    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"]
Jul 14 01:32:39.858107 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:32:39.858083    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7e931986_a1d8_4aee_8df0_3908ef964f5e.slice/crio-abf9c26e606896f6ac49e900ceed0341ae266a50960c14b04fba249e35ad84b1 WatchSource:0}: Error finding container abf9c26e606896f6ac49e900ceed0341ae266a50960c14b04fba249e35ad84b1: Status 404 returned error can't find the container with id abf9c26e606896f6ac49e900ceed0341ae266a50960c14b04fba249e35ad84b1
Jul 14 01:32:40.358111 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:40.358078    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz" event={"ID":"7e931986-a1d8-4aee-8df0-3908ef964f5e","Type":"ContainerStarted","Data":"a1834c136b20323a4442e6178ae1cbfeb311d3bcba4f0dc524c2056c930e970b"}
Jul 14 01:32:40.358111 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:40.358118    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz" event={"ID":"7e931986-a1d8-4aee-8df0-3908ef964f5e","Type":"ContainerStarted","Data":"abf9c26e606896f6ac49e900ceed0341ae266a50960c14b04fba249e35ad84b1"}
Jul 14 01:32:40.375343 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:40.375290    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz" podStartSLOduration=1.375275443 podStartE2EDuration="1.375275443s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:32:40.373374865 +0000 UTC m=+2910.354459527" watchObservedRunningTime="2026-07-14 01:32:40.375275443 +0000 UTC m=+2910.356360104"
Jul 14 01:32:43.426956 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:43.426919    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:32:49.729876 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:49.729838    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:49.730425 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:49.729891    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:49.742403 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:49.742380    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:32:50.417040 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:32:50.416996    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:33:18.884486 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:18.884453    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"]
Jul 14 01:33:18.889444 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:18.889423    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:18.893436 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:18.893413    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"kube-root-ca.crt\""
Jul 14 01:33:18.893578 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:18.893470    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"openshift-service-ca.crt\""
Jul 14 01:33:18.893876 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:18.893863    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"rate-test-model-8fd48b-kserve-self-signed-certs\""
Jul 14 01:33:18.893958 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:18.893940    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-8fd48b\"/\"default-dockercfg-fwpjf\""
Jul 14 01:33:18.906260 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:18.906237    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"]
Jul 14 01:33:19.023959 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.023928    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tls-certs\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.024125 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.023969    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-home\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.024125 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.023987    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-model-cache\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.024125 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.024083    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tmp-dir\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.024125 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.024116    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-dshm\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.024303 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.024134    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flxrp\" (UniqueName: \"kubernetes.io/projected/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-kube-api-access-flxrp\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.124775 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.124730    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tls-certs\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.124775 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.124784    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-home\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.125011 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.124802    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-model-cache\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.125011 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.124836    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tmp-dir\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.125011 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.124855    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-dshm\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.125011 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.124872    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flxrp\" (UniqueName: \"kubernetes.io/projected/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-kube-api-access-flxrp\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.125285 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.125263    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-home\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.125285 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.125278    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tmp-dir\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.125396 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.125302    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-model-cache\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.127050 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.127026    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-dshm\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.127210 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.127194    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tls-certs\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.132369 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.132349    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-flxrp\" (UniqueName: \"kubernetes.io/projected/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-kube-api-access-flxrp\") pod \"rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") " pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.198526 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.198420    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:19.326312 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.326292    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"]
Jul 14 01:33:19.328074 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:33:19.328048    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3450e9e8_3517_4ca2_9cd5_6daba2dd3897.slice/crio-febdfeffb4481fa2ca2b2b412ccaf43f4e09691a9926f27804c322dca6770160 WatchSource:0}: Error finding container febdfeffb4481fa2ca2b2b412ccaf43f4e09691a9926f27804c322dca6770160: Status 404 returned error can't find the container with id febdfeffb4481fa2ca2b2b412ccaf43f4e09691a9926f27804c322dca6770160
Jul 14 01:33:19.546436 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.546344    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr" event={"ID":"3450e9e8-3517-4ca2-9cd5-6daba2dd3897","Type":"ContainerStarted","Data":"7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1"}
Jul 14 01:33:19.546436 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.546385    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr" event={"ID":"3450e9e8-3517-4ca2-9cd5-6daba2dd3897","Type":"ContainerStarted","Data":"febdfeffb4481fa2ca2b2b412ccaf43f4e09691a9926f27804c322dca6770160"}
Jul 14 01:33:19.561901 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:19.561853    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr" podStartSLOduration=1.561836751 podStartE2EDuration="1.561836751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:33:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:33:19.560952735 +0000 UTC m=+2949.542037394" watchObservedRunningTime="2026-07-14 01:33:19.561836751 +0000 UTC m=+2949.542921412"
Jul 14 01:33:23.330992 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:23.330956    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:33:29.198859 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:29.198808    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:29.199373 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:29.198881    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:29.211239 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:29.211217    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:29.606868 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:29.606832    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:33:36.126676 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:36.126596    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:33:51.923648 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:51.923614    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:33:57.548888 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:57.548856    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"]
Jul 14 01:33:58.423174 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:33:58.423142    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"]
Jul 14 01:34:09.920015 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:09.919983    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:34:10.805130 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:10.805021    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:34:10.814104 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:10.814084    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:34:12.568159 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.568118    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz" podUID="7e931986-a1d8-4aee-8df0-3908ef964f5e" containerName="main" containerID="cri-o://a1834c136b20323a4442e6178ae1cbfeb311d3bcba4f0dc524c2056c930e970b" gracePeriod=45
Jul 14 01:34:12.808703 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.808672    2552 generic.go:356] "Generic (PLEG): container finished" podID="7e931986-a1d8-4aee-8df0-3908ef964f5e" containerID="a1834c136b20323a4442e6178ae1cbfeb311d3bcba4f0dc524c2056c930e970b" exitCode=0
Jul 14 01:34:12.808828 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.808740    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz" event={"ID":"7e931986-a1d8-4aee-8df0-3908ef964f5e","Type":"ContainerDied","Data":"a1834c136b20323a4442e6178ae1cbfeb311d3bcba4f0dc524c2056c930e970b"}
Jul 14 01:34:12.821349 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.821296    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:34:12.976410 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976380    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-dshm\") pod \"7e931986-a1d8-4aee-8df0-3908ef964f5e\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") "
Jul 14 01:34:12.976600 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976443    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-home\") pod \"7e931986-a1d8-4aee-8df0-3908ef964f5e\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") "
Jul 14 01:34:12.976600 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976477    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-tmp-dir\") pod \"7e931986-a1d8-4aee-8df0-3908ef964f5e\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") "
Jul 14 01:34:12.976600 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976549    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-model-cache\") pod \"7e931986-a1d8-4aee-8df0-3908ef964f5e\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") "
Jul 14 01:34:12.976767 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976604    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fft4r\" (UniqueName: \"kubernetes.io/projected/7e931986-a1d8-4aee-8df0-3908ef964f5e-kube-api-access-fft4r\") pod \"7e931986-a1d8-4aee-8df0-3908ef964f5e\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") "
Jul 14 01:34:12.976767 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976632    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e931986-a1d8-4aee-8df0-3908ef964f5e-tls-certs\") pod \"7e931986-a1d8-4aee-8df0-3908ef964f5e\" (UID: \"7e931986-a1d8-4aee-8df0-3908ef964f5e\") "
Jul 14 01:34:12.976767 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976731    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-home" (OuterVolumeSpecName: "home") pod "7e931986-a1d8-4aee-8df0-3908ef964f5e" (UID: "7e931986-a1d8-4aee-8df0-3908ef964f5e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:12.976904 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976808    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-model-cache" (OuterVolumeSpecName: "model-cache") pod "7e931986-a1d8-4aee-8df0-3908ef964f5e" (UID: "7e931986-a1d8-4aee-8df0-3908ef964f5e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:12.976904 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.976824    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "7e931986-a1d8-4aee-8df0-3908ef964f5e" (UID: "7e931986-a1d8-4aee-8df0-3908ef964f5e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:12.977028 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.977008    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:12.977073 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.977036    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:12.977073 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.977049    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:12.978575 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.978554    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-dshm" (OuterVolumeSpecName: "dshm") pod "7e931986-a1d8-4aee-8df0-3908ef964f5e" (UID: "7e931986-a1d8-4aee-8df0-3908ef964f5e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:12.978702 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.978682    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e931986-a1d8-4aee-8df0-3908ef964f5e-kube-api-access-fft4r" (OuterVolumeSpecName: "kube-api-access-fft4r") pod "7e931986-a1d8-4aee-8df0-3908ef964f5e" (UID: "7e931986-a1d8-4aee-8df0-3908ef964f5e"). InnerVolumeSpecName "kube-api-access-fft4r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:12.978755 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:12.978733    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e931986-a1d8-4aee-8df0-3908ef964f5e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "7e931986-a1d8-4aee-8df0-3908ef964f5e" (UID: "7e931986-a1d8-4aee-8df0-3908ef964f5e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:34:13.078013 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.077929    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fft4r\" (UniqueName: \"kubernetes.io/projected/7e931986-a1d8-4aee-8df0-3908ef964f5e-kube-api-access-fft4r\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.078013 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.077959    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7e931986-a1d8-4aee-8df0-3908ef964f5e-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.078013 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.077969    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7e931986-a1d8-4aee-8df0-3908ef964f5e-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.443258 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.443157    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr" podUID="3450e9e8-3517-4ca2-9cd5-6daba2dd3897" containerName="main" containerID="cri-o://7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1" gracePeriod=45
Jul 14 01:34:13.690481 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.690458    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:34:13.784990 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.784898    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-model-cache\") pod \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") "
Jul 14 01:34:13.784990 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.784942    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tls-certs\") pod \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") "
Jul 14 01:34:13.785218 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785075    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tmp-dir\") pod \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") "
Jul 14 01:34:13.785218 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785091    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-dshm\") pod \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") "
Jul 14 01:34:13.785218 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785110    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-home\") pod \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") "
Jul 14 01:34:13.785218 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785125    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-flxrp\" (UniqueName: \"kubernetes.io/projected/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-kube-api-access-flxrp\") pod \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\" (UID: \"3450e9e8-3517-4ca2-9cd5-6daba2dd3897\") "
Jul 14 01:34:13.785426 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785220    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-model-cache" (OuterVolumeSpecName: "model-cache") pod "3450e9e8-3517-4ca2-9cd5-6daba2dd3897" (UID: "3450e9e8-3517-4ca2-9cd5-6daba2dd3897"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:13.785426 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785348    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-home" (OuterVolumeSpecName: "home") pod "3450e9e8-3517-4ca2-9cd5-6daba2dd3897" (UID: "3450e9e8-3517-4ca2-9cd5-6daba2dd3897"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:13.785426 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785400    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-model-cache\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.785570 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.785462    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3450e9e8-3517-4ca2-9cd5-6daba2dd3897" (UID: "3450e9e8-3517-4ca2-9cd5-6daba2dd3897"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:13.787327 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.787297    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-kube-api-access-flxrp" (OuterVolumeSpecName: "kube-api-access-flxrp") pod "3450e9e8-3517-4ca2-9cd5-6daba2dd3897" (UID: "3450e9e8-3517-4ca2-9cd5-6daba2dd3897"). InnerVolumeSpecName "kube-api-access-flxrp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:13.787447 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.787305    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3450e9e8-3517-4ca2-9cd5-6daba2dd3897" (UID: "3450e9e8-3517-4ca2-9cd5-6daba2dd3897"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:34:13.787447 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.787339    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-dshm" (OuterVolumeSpecName: "dshm") pod "3450e9e8-3517-4ca2-9cd5-6daba2dd3897" (UID: "3450e9e8-3517-4ca2-9cd5-6daba2dd3897"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 14 01:34:13.814760 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.814732    2552 generic.go:356] "Generic (PLEG): container finished" podID="3450e9e8-3517-4ca2-9cd5-6daba2dd3897" containerID="7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1" exitCode=0
Jul 14 01:34:13.814902 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.814775    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"
Jul 14 01:34:13.814902 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.814810    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr" event={"ID":"3450e9e8-3517-4ca2-9cd5-6daba2dd3897","Type":"ContainerDied","Data":"7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1"}
Jul 14 01:34:13.814902 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.814849    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr" event={"ID":"3450e9e8-3517-4ca2-9cd5-6daba2dd3897","Type":"ContainerDied","Data":"febdfeffb4481fa2ca2b2b412ccaf43f4e09691a9926f27804c322dca6770160"}
Jul 14 01:34:13.814902 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.814865    2552 scope.go:117] "RemoveContainer" containerID="7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1"
Jul 14 01:34:13.816171 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.816149    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz" event={"ID":"7e931986-a1d8-4aee-8df0-3908ef964f5e","Type":"ContainerDied","Data":"abf9c26e606896f6ac49e900ceed0341ae266a50960c14b04fba249e35ad84b1"}
Jul 14 01:34:13.816287 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.816176    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"
Jul 14 01:34:13.824391 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.824371    2552 scope.go:117] "RemoveContainer" containerID="7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1"
Jul 14 01:34:13.824704 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:34:13.824665    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1\": container with ID starting with 7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1 not found: ID does not exist" containerID="7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1"
Jul 14 01:34:13.824704 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.824695    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1"} err="failed to get container status \"7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1\": rpc error: code = NotFound desc = could not find container \"7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1\": container with ID starting with 7d346686d3c57172e510d5d60036e09f2ce61cc59b95d8a195eab5b48038e4e1 not found: ID does not exist"
Jul 14 01:34:13.824816 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.824710    2552 scope.go:117] "RemoveContainer" containerID="a1834c136b20323a4442e6178ae1cbfeb311d3bcba4f0dc524c2056c930e970b"
Jul 14 01:34:13.839310 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.839287    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"]
Jul 14 01:34:13.842763 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.842746    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-8fd48b/rate-test-model-8fd48b-kserve-6d9c7dd5b7-n4pgr"]
Jul 14 01:34:13.852722 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.852699    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"]
Jul 14 01:34:13.855563 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.855544    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-99f60e/rate-test-model-99f60e-kserve-6cbb87fbcf-2vvcz"]
Jul 14 01:34:13.886694 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.886670    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tmp-dir\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.886694 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.886691    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-dshm\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.886694 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.886700    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-home\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.886928 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.886708    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flxrp\" (UniqueName: \"kubernetes.io/projected/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-kube-api-access-flxrp\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:13.886928 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:13.886718    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3450e9e8-3517-4ca2-9cd5-6daba2dd3897-tls-certs\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:14.433933 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.433904    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/test-curl-7710-140225656771616"]
Jul 14 01:34:14.434240 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.434228    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3450e9e8-3517-4ca2-9cd5-6daba2dd3897" containerName="main"
Jul 14 01:34:14.434283 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.434241    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3450e9e8-3517-4ca2-9cd5-6daba2dd3897" containerName="main"
Jul 14 01:34:14.434283 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.434255    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7e931986-a1d8-4aee-8df0-3908ef964f5e" containerName="main"
Jul 14 01:34:14.434283 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.434260    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7e931986-a1d8-4aee-8df0-3908ef964f5e" containerName="main"
Jul 14 01:34:14.434373 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.434315    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3450e9e8-3517-4ca2-9cd5-6daba2dd3897" containerName="main"
Jul 14 01:34:14.434373 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.434324    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7e931986-a1d8-4aee-8df0-3908ef964f5e" containerName="main"
Jul 14 01:34:14.438596 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.438580    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:14.441282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.441233    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"default-dockercfg-jw4fv\""
Jul 14 01:34:14.442121 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.442097    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7710-140225656771616"]
Jul 14 01:34:14.525623 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.525598    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3450e9e8-3517-4ca2-9cd5-6daba2dd3897" path="/var/lib/kubelet/pods/3450e9e8-3517-4ca2-9cd5-6daba2dd3897/volumes"
Jul 14 01:34:14.525977 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.525964    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7e931986-a1d8-4aee-8df0-3908ef964f5e" path="/var/lib/kubelet/pods/7e931986-a1d8-4aee-8df0-3908ef964f5e/volumes"
Jul 14 01:34:14.591960 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.591937    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dd9k\" (UniqueName: \"kubernetes.io/projected/80f75d1f-edc7-480d-aa1b-81617ff07549-kube-api-access-2dd9k\") pod \"test-curl-7710-140225656771616\" (UID: \"80f75d1f-edc7-480d-aa1b-81617ff07549\") " pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:14.693229 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.693170    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dd9k\" (UniqueName: \"kubernetes.io/projected/80f75d1f-edc7-480d-aa1b-81617ff07549-kube-api-access-2dd9k\") pod \"test-curl-7710-140225656771616\" (UID: \"80f75d1f-edc7-480d-aa1b-81617ff07549\") " pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:14.700638 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.700618    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dd9k\" (UniqueName: \"kubernetes.io/projected/80f75d1f-edc7-480d-aa1b-81617ff07549-kube-api-access-2dd9k\") pod \"test-curl-7710-140225656771616\" (UID: \"80f75d1f-edc7-480d-aa1b-81617ff07549\") " pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:14.748300 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.748280    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:14.872201 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.872172    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/test-curl-7710-140225656771616"]
Jul 14 01:34:14.874897 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:34:14.874859    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod80f75d1f_edc7_480d_aa1b_81617ff07549.slice/crio-0627f87d46c9ad8dac7ba819b7391d5d93d1e52bc5fb6c635fe143752fac28ba WatchSource:0}: Error finding container 0627f87d46c9ad8dac7ba819b7391d5d93d1e52bc5fb6c635fe143752fac28ba: Status 404 returned error can't find the container with id 0627f87d46c9ad8dac7ba819b7391d5d93d1e52bc5fb6c635fe143752fac28ba
Jul 14 01:34:14.876274 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:14.876255    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 14 01:34:15.836247 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:15.836221    2552 generic.go:356] "Generic (PLEG): container finished" podID="80f75d1f-edc7-480d-aa1b-81617ff07549" containerID="a8820d851c5a82dca9b4d7d3a51fcef71e1042dfeb10f605eeddfeb9256746e3" exitCode=0
Jul 14 01:34:15.836578 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:15.836257    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7710-140225656771616" event={"ID":"80f75d1f-edc7-480d-aa1b-81617ff07549","Type":"ContainerDied","Data":"a8820d851c5a82dca9b4d7d3a51fcef71e1042dfeb10f605eeddfeb9256746e3"}
Jul 14 01:34:15.836578 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:15.836285    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/test-curl-7710-140225656771616" event={"ID":"80f75d1f-edc7-480d-aa1b-81617ff07549","Type":"ContainerStarted","Data":"0627f87d46c9ad8dac7ba819b7391d5d93d1e52bc5fb6c635fe143752fac28ba"}
Jul 14 01:34:16.986122 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:16.986098    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:17.017875 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.017853    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_test-curl-7710-140225656771616_80f75d1f-edc7-480d-aa1b-81617ff07549/test-curl-7710-140225656771616/0.log"
Jul 14 01:34:17.039569 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.039545    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/test-curl-7710-140225656771616"]
Jul 14 01:34:17.044241 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.044216    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/test-curl-7710-140225656771616"]
Jul 14 01:34:17.111970 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.111892    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dd9k\" (UniqueName: \"kubernetes.io/projected/80f75d1f-edc7-480d-aa1b-81617ff07549-kube-api-access-2dd9k\") pod \"80f75d1f-edc7-480d-aa1b-81617ff07549\" (UID: \"80f75d1f-edc7-480d-aa1b-81617ff07549\") "
Jul 14 01:34:17.113984 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.113961    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80f75d1f-edc7-480d-aa1b-81617ff07549-kube-api-access-2dd9k" (OuterVolumeSpecName: "kube-api-access-2dd9k") pod "80f75d1f-edc7-480d-aa1b-81617ff07549" (UID: "80f75d1f-edc7-480d-aa1b-81617ff07549"). InnerVolumeSpecName "kube-api-access-2dd9k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:17.212794 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.212760    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dd9k\" (UniqueName: \"kubernetes.io/projected/80f75d1f-edc7-480d-aa1b-81617ff07549-kube-api-access-2dd9k\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:17.846672 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.846644    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0627f87d46c9ad8dac7ba819b7391d5d93d1e52bc5fb6c635fe143752fac28ba"
Jul 14 01:34:17.846672 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:17.846668    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:17.856216 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:34:17.856190    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7710-140225656771616\" is forbidden: User \"system:node:ip-10-0-129-125.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"openshift-ingress\": no relationship found between node 'ip-10-0-129-125.ec2.internal' and this object" podUID="80f75d1f-edc7-480d-aa1b-81617ff07549" pod="openshift-ingress/test-curl-7710-140225656771616"
Jul 14 01:34:18.524553 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:18.524524    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80f75d1f-edc7-480d-aa1b-81617ff07549" path="/var/lib/kubelet/pods/80f75d1f-edc7-480d-aa1b-81617ff07549/volumes"
Jul 14 01:34:27.630062 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:27.630032    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"]
Jul 14 01:34:27.630530 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:27.630248    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd" podUID="c0d7704d-57cb-44aa-8123-cd2d27bda47e" containerName="maas-api" containerID="cri-o://4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe" gracePeriod=30
Jul 14 01:34:28.077916 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.077889    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:34:28.102253 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.102222    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d7704d-57cb-44aa-8123-cd2d27bda47e-openshift-service-ca\") pod \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") "
Jul 14 01:34:28.102406 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.102268    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw9lr\" (UniqueName: \"kubernetes.io/projected/c0d7704d-57cb-44aa-8123-cd2d27bda47e-kube-api-access-dw9lr\") pod \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") "
Jul 14 01:34:28.102406 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.102287    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d7704d-57cb-44aa-8123-cd2d27bda47e-maas-api-tls\") pod \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\" (UID: \"c0d7704d-57cb-44aa-8123-cd2d27bda47e\") "
Jul 14 01:34:28.102680 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.102654    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d7704d-57cb-44aa-8123-cd2d27bda47e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "c0d7704d-57cb-44aa-8123-cd2d27bda47e" (UID: "c0d7704d-57cb-44aa-8123-cd2d27bda47e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:34:28.104458 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.104425    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d7704d-57cb-44aa-8123-cd2d27bda47e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "c0d7704d-57cb-44aa-8123-cd2d27bda47e" (UID: "c0d7704d-57cb-44aa-8123-cd2d27bda47e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:34:28.104604 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.104519    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d7704d-57cb-44aa-8123-cd2d27bda47e-kube-api-access-dw9lr" (OuterVolumeSpecName: "kube-api-access-dw9lr") pod "c0d7704d-57cb-44aa-8123-cd2d27bda47e" (UID: "c0d7704d-57cb-44aa-8123-cd2d27bda47e"). InnerVolumeSpecName "kube-api-access-dw9lr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:28.203653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.203563    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/c0d7704d-57cb-44aa-8123-cd2d27bda47e-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:28.203653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.203597    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/c0d7704d-57cb-44aa-8123-cd2d27bda47e-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:28.203653 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.203607    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw9lr\" (UniqueName: \"kubernetes.io/projected/c0d7704d-57cb-44aa-8123-cd2d27bda47e-kube-api-access-dw9lr\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:28.898794 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.898762    2552 generic.go:356] "Generic (PLEG): container finished" podID="c0d7704d-57cb-44aa-8123-cd2d27bda47e" containerID="4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe" exitCode=0
Jul 14 01:34:28.899299 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.898804    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"
Jul 14 01:34:28.899299 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.898815    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd" event={"ID":"c0d7704d-57cb-44aa-8123-cd2d27bda47e","Type":"ContainerDied","Data":"4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe"}
Jul 14 01:34:28.899299 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.898858    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd" event={"ID":"c0d7704d-57cb-44aa-8123-cd2d27bda47e","Type":"ContainerDied","Data":"a0aff1122f7f201dba315aac3a13537f0969393d3d06c01b61dee31c5ce13f62"}
Jul 14 01:34:28.899299 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.898883    2552 scope.go:117] "RemoveContainer" containerID="4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe"
Jul 14 01:34:28.907370 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.907351    2552 scope.go:117] "RemoveContainer" containerID="4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe"
Jul 14 01:34:28.907658 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:34:28.907638    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe\": container with ID starting with 4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe not found: ID does not exist" containerID="4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe"
Jul 14 01:34:28.907730 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.907670    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe"} err="failed to get container status \"4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe\": rpc error: code = NotFound desc = could not find container \"4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe\": container with ID starting with 4c9d8e0ff1a0f61e8be06254cf592c7e4c5b678b37617b0496cd6475e1d4d0fe not found: ID does not exist"
Jul 14 01:34:28.913807 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.913782    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"]
Jul 14 01:34:28.917545 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:28.917519    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-99f60e-7b4f75b6c8-wr9sd"]
Jul 14 01:34:30.523694 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:30.523663    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c0d7704d-57cb-44aa-8123-cd2d27bda47e" path="/var/lib/kubelet/pods/c0d7704d-57cb-44aa-8123-cd2d27bda47e/volumes"
Jul 14 01:34:30.922171 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:30.922137    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:34:43.386130 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.386097    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"]
Jul 14 01:34:43.386468 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.386416    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c0d7704d-57cb-44aa-8123-cd2d27bda47e" containerName="maas-api"
Jul 14 01:34:43.386468 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.386426    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c0d7704d-57cb-44aa-8123-cd2d27bda47e" containerName="maas-api"
Jul 14 01:34:43.386468 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.386440    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80f75d1f-edc7-480d-aa1b-81617ff07549" containerName="test-curl-7710-140225656771616"
Jul 14 01:34:43.386468 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.386447    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="80f75d1f-edc7-480d-aa1b-81617ff07549" containerName="test-curl-7710-140225656771616"
Jul 14 01:34:43.386630 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.386519    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="80f75d1f-edc7-480d-aa1b-81617ff07549" containerName="test-curl-7710-140225656771616"
Jul 14 01:34:43.386630 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.386531    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c0d7704d-57cb-44aa-8123-cd2d27bda47e" containerName="maas-api"
Jul 14 01:34:43.390906 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.390889    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"
Jul 14 01:34:43.393403 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.393374    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-kl2pl\""
Jul 14 01:34:43.395958 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.395938    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"]
Jul 14 01:34:43.437116 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.437091    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07d36-e50e-4b42-8229-0af18231cbb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968\" (UID: \"c4c07d36-e50e-4b42-8229-0af18231cbb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"
Jul 14 01:34:43.538188 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.538161    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07d36-e50e-4b42-8229-0af18231cbb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968\" (UID: \"c4c07d36-e50e-4b42-8229-0af18231cbb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"
Jul 14 01:34:43.538724 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.538708    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07d36-e50e-4b42-8229-0af18231cbb0-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968\" (UID: \"c4c07d36-e50e-4b42-8229-0af18231cbb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"
Jul 14 01:34:43.702458 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.702383    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"
Jul 14 01:34:43.829067 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.829045    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"]
Jul 14 01:34:43.830994 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:34:43.830969    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4c07d36_e50e_4b42_8229_0af18231cbb0.slice/crio-5562ade486171898956ee283c5264b6a1089dc5a3cb2cf3984918b210f680d35 WatchSource:0}: Error finding container 5562ade486171898956ee283c5264b6a1089dc5a3cb2cf3984918b210f680d35: Status 404 returned error can't find the container with id 5562ade486171898956ee283c5264b6a1089dc5a3cb2cf3984918b210f680d35
Jul 14 01:34:43.976541 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:43.976467    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968" event={"ID":"c4c07d36-e50e-4b42-8229-0af18231cbb0","Type":"ContainerStarted","Data":"5562ade486171898956ee283c5264b6a1089dc5a3cb2cf3984918b210f680d35"}
Jul 14 01:34:44.982267 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:44.982230    2552 generic.go:356] "Generic (PLEG): container finished" podID="c4c07d36-e50e-4b42-8229-0af18231cbb0" containerID="dd21e5563885c42094eb7babe7faf9b8f98206a2fa4f6b0ad0f63a720707c402" exitCode=0
Jul 14 01:34:44.982724 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:44.982318    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968" event={"ID":"c4c07d36-e50e-4b42-8229-0af18231cbb0","Type":"ContainerDied","Data":"dd21e5563885c42094eb7babe7faf9b8f98206a2fa4f6b0ad0f63a720707c402"}
Jul 14 01:34:46.106793 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:46.106765    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"
Jul 14 01:34:46.160439 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:46.160417    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07d36-e50e-4b42-8229-0af18231cbb0-maas-api-service-ca\") pod \"c4c07d36-e50e-4b42-8229-0af18231cbb0\" (UID: \"c4c07d36-e50e-4b42-8229-0af18231cbb0\") "
Jul 14 01:34:46.160750 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:46.160729    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4c07d36-e50e-4b42-8229-0af18231cbb0-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c4c07d36-e50e-4b42-8229-0af18231cbb0" (UID: "c4c07d36-e50e-4b42-8229-0af18231cbb0"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:34:46.261516 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:46.261424    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c4c07d36-e50e-4b42-8229-0af18231cbb0-maas-api-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:46.997013 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:46.996981    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968" event={"ID":"c4c07d36-e50e-4b42-8229-0af18231cbb0","Type":"ContainerDied","Data":"5562ade486171898956ee283c5264b6a1089dc5a3cb2cf3984918b210f680d35"}
Jul 14 01:34:46.997013 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:46.997010    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968"
Jul 14 01:34:46.997242 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:46.997031    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5562ade486171898956ee283c5264b6a1089dc5a3cb2cf3984918b210f680d35"
Jul 14 01:34:53.770746 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:53.770714    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"]
Jul 14 01:34:53.771127 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:53.770939    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" podUID="d4ad474c-77ff-40a2-ab33-b752ae1a967d" containerName="maas-api" containerID="cri-o://5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d" gracePeriod=30
Jul 14 01:34:54.097382 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.097343    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" podUID="d4ad474c-77ff-40a2-ab33-b752ae1a967d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.0.50:8443/health\": dial tcp 10.133.0.50:8443: connect: connection refused"
Jul 14 01:34:54.223744 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.223720    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:34:54.328129 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.328108    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-62s8j\" (UniqueName: \"kubernetes.io/projected/d4ad474c-77ff-40a2-ab33-b752ae1a967d-kube-api-access-62s8j\") pod \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") "
Jul 14 01:34:54.328270 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.328197    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls\") pod \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") "
Jul 14 01:34:54.328270 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.328240    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ad474c-77ff-40a2-ab33-b752ae1a967d-openshift-service-ca\") pod \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\" (UID: \"d4ad474c-77ff-40a2-ab33-b752ae1a967d\") "
Jul 14 01:34:54.328656 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.328633    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4ad474c-77ff-40a2-ab33-b752ae1a967d-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d4ad474c-77ff-40a2-ab33-b752ae1a967d" (UID: "d4ad474c-77ff-40a2-ab33-b752ae1a967d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 14 01:34:54.330172 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.330139    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d4ad474c-77ff-40a2-ab33-b752ae1a967d" (UID: "d4ad474c-77ff-40a2-ab33-b752ae1a967d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 14 01:34:54.330268 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.330187    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4ad474c-77ff-40a2-ab33-b752ae1a967d-kube-api-access-62s8j" (OuterVolumeSpecName: "kube-api-access-62s8j") pod "d4ad474c-77ff-40a2-ab33-b752ae1a967d" (UID: "d4ad474c-77ff-40a2-ab33-b752ae1a967d"). InnerVolumeSpecName "kube-api-access-62s8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 14 01:34:54.429514 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.429479    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4ad474c-77ff-40a2-ab33-b752ae1a967d-openshift-service-ca\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:54.429514 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.429516    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-62s8j\" (UniqueName: \"kubernetes.io/projected/d4ad474c-77ff-40a2-ab33-b752ae1a967d-kube-api-access-62s8j\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:54.429647 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:54.429528    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d4ad474c-77ff-40a2-ab33-b752ae1a967d-maas-api-tls\") on node \"ip-10-0-129-125.ec2.internal\" DevicePath \"\""
Jul 14 01:34:55.035680 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.035642    2552 generic.go:356] "Generic (PLEG): container finished" podID="d4ad474c-77ff-40a2-ab33-b752ae1a967d" containerID="5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d" exitCode=0
Jul 14 01:34:55.036059 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.035689    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"
Jul 14 01:34:55.036059 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.035730    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" event={"ID":"d4ad474c-77ff-40a2-ab33-b752ae1a967d","Type":"ContainerDied","Data":"5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d"}
Jul 14 01:34:55.036059 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.035787    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh" event={"ID":"d4ad474c-77ff-40a2-ab33-b752ae1a967d","Type":"ContainerDied","Data":"5c8c3ba4aed43f61b366191a16f5e3130cba63c5b27f9e0e3b34e3976564cff4"}
Jul 14 01:34:55.036059 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.035808    2552 scope.go:117] "RemoveContainer" containerID="5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d"
Jul 14 01:34:55.044179 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.044162    2552 scope.go:117] "RemoveContainer" containerID="5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d"
Jul 14 01:34:55.044403 ip-10-0-129-125 kubenswrapper[2552]: E0714 01:34:55.044385    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d\": container with ID starting with 5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d not found: ID does not exist" containerID="5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d"
Jul 14 01:34:55.044463 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.044409    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d"} err="failed to get container status \"5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d\": rpc error: code = NotFound desc = could not find container \"5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d\": container with ID starting with 5909c92e5dd081d01c1362031b9ceb65624065c905a788e054914eb3171cff1d not found: ID does not exist"
Jul 14 01:34:55.052757 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.052731    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"]
Jul 14 01:34:55.054397 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:55.054378    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-8fd48b-6c4ffd5d88-p8pbh"]
Jul 14 01:34:56.524152 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:56.524120    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d4ad474c-77ff-40a2-ab33-b752ae1a967d" path="/var/lib/kubelet/pods/d4ad474c-77ff-40a2-ab33-b752ae1a967d/volumes"
Jul 14 01:34:56.728906 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:34:56.728875    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-54d6d988d5-b9zjq"]
Jul 14 01:35:15.641925 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:15.641897    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7d6985f7-s4wp8_1eb3d285-aa44-4bf9-b85b-81ed74cb1be1/copy-manifests/0.log"
Jul 14 01:35:15.659218 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:15.659187    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7d6985f7-s4wp8_1eb3d285-aa44-4bf9-b85b-81ed74cb1be1/manager/0.log"
Jul 14 01:35:16.292889 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:16.292856    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29733195-qsg95_60240235-8aa5-4712-9ca3-30ab4ddb7d4d/cleanup/0.log"
Jul 14 01:35:16.517674 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:16.517635    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-b9a4f161-knx78_ff072b7b-0a3e-48cc-bedc-823661c7ceb2/revoke-keys/0.log"
Jul 14 01:35:16.630812 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:16.630784    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-ait-f0fe996a-xlxxj_b3a8b3ac-a1ee-4f32-b2bc-0dfdd57dfdc4/revoke-keys/0.log"
Jul 14 01:35:16.852391 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:16.852363    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-api-b-e7c477-zbl8z_318df0d3-8a01-44bd-a932-a79b75b71323/revoke-keys/0.log"
Jul 14 01:35:16.973549 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:16.973449    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-derive-b3a22390-tvjsq_68ed5ba4-b187-46ce-bda5-4bcd57b5cf6d/revoke-keys/0.log"
Jul 14 01:35:17.231998 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:17.231918    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-inf-b-cc490a-z6ll2_bb3d5dd0-7825-4d69-b806-80a5c7052bdd/revoke-keys/0.log"
Jul 14 01:35:17.346217 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:17.346179    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-migrate-6969890f-5qx7q_3a7fd887-bd80-4198-b758-ca59fbdc9550/revoke-keys/0.log"
Jul 14 01:35:17.460689 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:17.460658    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-2a62f7c8-khkcc_a4ea6b3a-408b-4965-8bd2-42024ba3217d/revoke-keys/0.log"
Jul 14 01:35:17.575211 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:17.575185    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-mt-48d1e8ae-wcclb_6be7c25f-9e48-4a28-8974-45005a7080ca/revoke-keys/0.log"
Jul 14 01:35:17.803005 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:17.802978    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-keys-e2e-shared-b-8fd48b-gb968_c4c07d36-e50e-4b42-8229-0af18231cbb0/revoke-keys/0.log"
Jul 14 01:35:19.438399 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:19.438356    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-9wv7m_a7312b34-36e0-49cc-83b2-f4f5ece254e0/manager/0.log"
Jul 14 01:35:19.547766 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:19.547739    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nzhvx_adb80f30-3030-45b7-80b0-f74b99667170/kuadrant-console-plugin/0.log"
Jul 14 01:35:19.920996 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:19.920971    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-b9zjq_7199c424-9674-4941-84e1-1e072b30ea67/limitador/0.log"
Jul 14 01:35:20.759351 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:20.759326    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f876ccc94-wrs6n_93bb0628-6a62-4442-a27e-bd8e614434b5/kube-auth-proxy/0.log"
Jul 14 01:35:22.076996 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:22.076969    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-dxfrg_7bdab809-df29-4923-a9f3-bb4487b83069/main/0.log"
Jul 14 01:35:26.241007 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.240974    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nwk5x/must-gather-m5lxz"]
Jul 14 01:35:26.241360 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.241339    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d4ad474c-77ff-40a2-ab33-b752ae1a967d" containerName="maas-api"
Jul 14 01:35:26.241360 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.241350    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d4ad474c-77ff-40a2-ab33-b752ae1a967d" containerName="maas-api"
Jul 14 01:35:26.241437 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.241368    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4c07d36-e50e-4b42-8229-0af18231cbb0" containerName="revoke-keys"
Jul 14 01:35:26.241437 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.241374    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4c07d36-e50e-4b42-8229-0af18231cbb0" containerName="revoke-keys"
Jul 14 01:35:26.241437 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.241436    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d4ad474c-77ff-40a2-ab33-b752ae1a967d" containerName="maas-api"
Jul 14 01:35:26.241560 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.241447    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4c07d36-e50e-4b42-8229-0af18231cbb0" containerName="revoke-keys"
Jul 14 01:35:26.244584 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.244567    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.246992 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.246967    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwk5x\"/\"kube-root-ca.crt\""
Jul 14 01:35:26.247113 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.246974    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-nwk5x\"/\"default-dockercfg-zsgk7\""
Jul 14 01:35:26.247113 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.247043    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-nwk5x\"/\"openshift-service-ca.crt\""
Jul 14 01:35:26.254775 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.254753    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwk5x/must-gather-m5lxz"]
Jul 14 01:35:26.287221 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.287197    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3dd702f7-8355-4edb-af08-eb127ecef8db-must-gather-output\") pod \"must-gather-m5lxz\" (UID: \"3dd702f7-8355-4edb-af08-eb127ecef8db\") " pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.287311 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.287242    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2xk58\" (UniqueName: \"kubernetes.io/projected/3dd702f7-8355-4edb-af08-eb127ecef8db-kube-api-access-2xk58\") pod \"must-gather-m5lxz\" (UID: \"3dd702f7-8355-4edb-af08-eb127ecef8db\") " pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.387713 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.387687    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3dd702f7-8355-4edb-af08-eb127ecef8db-must-gather-output\") pod \"must-gather-m5lxz\" (UID: \"3dd702f7-8355-4edb-af08-eb127ecef8db\") " pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.387807 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.387725    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2xk58\" (UniqueName: \"kubernetes.io/projected/3dd702f7-8355-4edb-af08-eb127ecef8db-kube-api-access-2xk58\") pod \"must-gather-m5lxz\" (UID: \"3dd702f7-8355-4edb-af08-eb127ecef8db\") " pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.388019 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.388001    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/3dd702f7-8355-4edb-af08-eb127ecef8db-must-gather-output\") pod \"must-gather-m5lxz\" (UID: \"3dd702f7-8355-4edb-af08-eb127ecef8db\") " pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.398255 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.398236    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2xk58\" (UniqueName: \"kubernetes.io/projected/3dd702f7-8355-4edb-af08-eb127ecef8db-kube-api-access-2xk58\") pod \"must-gather-m5lxz\" (UID: \"3dd702f7-8355-4edb-af08-eb127ecef8db\") " pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.553550 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.553461    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwk5x/must-gather-m5lxz"
Jul 14 01:35:26.674417 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:26.674393    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwk5x/must-gather-m5lxz"]
Jul 14 01:35:26.676355 ip-10-0-129-125 kubenswrapper[2552]: W0714 01:35:26.676322    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3dd702f7_8355_4edb_af08_eb127ecef8db.slice/crio-f62dc30a69eff9dc4b7d15f47522c96c2d8a66f5f9e1168e0b7c607b0defa4e4 WatchSource:0}: Error finding container f62dc30a69eff9dc4b7d15f47522c96c2d8a66f5f9e1168e0b7c607b0defa4e4: Status 404 returned error can't find the container with id f62dc30a69eff9dc4b7d15f47522c96c2d8a66f5f9e1168e0b7c607b0defa4e4
Jul 14 01:35:27.196560 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:27.196465    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwk5x/must-gather-m5lxz" event={"ID":"3dd702f7-8355-4edb-af08-eb127ecef8db","Type":"ContainerStarted","Data":"f62dc30a69eff9dc4b7d15f47522c96c2d8a66f5f9e1168e0b7c607b0defa4e4"}
Jul 14 01:35:28.206222 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:28.206182    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwk5x/must-gather-m5lxz" event={"ID":"3dd702f7-8355-4edb-af08-eb127ecef8db","Type":"ContainerStarted","Data":"636749a6ab3dd872dc5a8564c1130dcf16898d2a1d707ea28008049c550ae0c2"}
Jul 14 01:35:28.206704 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:28.206242    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwk5x/must-gather-m5lxz" event={"ID":"3dd702f7-8355-4edb-af08-eb127ecef8db","Type":"ContainerStarted","Data":"cadc08a4cbdc43e92f0b793159055bb8f9f52b5c26b27e419d77aebbc7773c04"}
Jul 14 01:35:28.220809 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:28.220753    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nwk5x/must-gather-m5lxz" podStartSLOduration=1.398492824 podStartE2EDuration="2.220735851s" totalImagesPullingTime="822.243027ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-14 01:35:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:35:28.22034961 +0000 UTC m=+3078.201434272" watchObservedRunningTime="2026-07-14 01:35:28.220735851 +0000 UTC m=+3078.201820515"
Jul 14 01:35:29.020311 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:29.020283    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w5xkv_d727cbd0-53be-43d2-8798-f91cf7d73d0a/global-pull-secret-syncer/0.log"
Jul 14 01:35:29.149399 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:29.149370    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-xx6fm_bae5b906-8455-4113-ba0b-5846d7b67f9e/konnectivity-agent/0.log"
Jul 14 01:35:29.170934 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:29.170903    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-125.ec2.internal_e7e5f961bb90f8432e41003971876c05/haproxy/0.log"
Jul 14 01:35:34.759112 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:34.759074    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-9wv7m_a7312b34-36e0-49cc-83b2-f4f5ece254e0/manager/0.log"
Jul 14 01:35:34.781263 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:34.781234    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66b5774985-nzhvx_adb80f30-3030-45b7-80b0-f74b99667170/kuadrant-console-plugin/0.log"
Jul 14 01:35:35.013071 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:35.012987    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-54d6d988d5-b9zjq_7199c424-9674-4941-84e1-1e072b30ea67/limitador/0.log"
Jul 14 01:35:36.710426 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:36.710391    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f866da78-6c90-4a3a-83cd-3afed9cfd0a9/alertmanager/0.log"
Jul 14 01:35:36.728597 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:36.728564    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f866da78-6c90-4a3a-83cd-3afed9cfd0a9/config-reloader/0.log"
Jul 14 01:35:36.749401 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:36.749328    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f866da78-6c90-4a3a-83cd-3afed9cfd0a9/kube-rbac-proxy-web/0.log"
Jul 14 01:35:36.767606 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:36.767575    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f866da78-6c90-4a3a-83cd-3afed9cfd0a9/kube-rbac-proxy/0.log"
Jul 14 01:35:36.786453 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:36.786423    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f866da78-6c90-4a3a-83cd-3afed9cfd0a9/kube-rbac-proxy-metric/0.log"
Jul 14 01:35:36.804410 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:36.804378    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f866da78-6c90-4a3a-83cd-3afed9cfd0a9/prom-label-proxy/0.log"
Jul 14 01:35:36.822643 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:36.822611    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_f866da78-6c90-4a3a-83cd-3afed9cfd0a9/init-config-reloader/0.log"
Jul 14 01:35:37.129266 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.129236    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p4trm_752e8910-0a6b-46b0-bc78-819374a99f62/node-exporter/0.log"
Jul 14 01:35:37.146082 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.146041    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p4trm_752e8910-0a6b-46b0-bc78-819374a99f62/kube-rbac-proxy/0.log"
Jul 14 01:35:37.165007 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.164970    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-p4trm_752e8910-0a6b-46b0-bc78-819374a99f62/init-textfile/0.log"
Jul 14 01:35:37.261396 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.261368    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_452bd4ac-95e3-458c-830e-547c32d88ee0/prometheus/0.log"
Jul 14 01:35:37.276900 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.276829    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_452bd4ac-95e3-458c-830e-547c32d88ee0/config-reloader/0.log"
Jul 14 01:35:37.301204 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.301172    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_452bd4ac-95e3-458c-830e-547c32d88ee0/thanos-sidecar/0.log"
Jul 14 01:35:37.318115 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.318090    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_452bd4ac-95e3-458c-830e-547c32d88ee0/kube-rbac-proxy-web/0.log"
Jul 14 01:35:37.334774 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.334736    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_452bd4ac-95e3-458c-830e-547c32d88ee0/kube-rbac-proxy/0.log"
Jul 14 01:35:37.352494 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.352470    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_452bd4ac-95e3-458c-830e-547c32d88ee0/kube-rbac-proxy-thanos/0.log"
Jul 14 01:35:37.369456 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.369433    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_452bd4ac-95e3-458c-830e-547c32d88ee0/init-config-reloader/0.log"
Jul 14 01:35:37.394676 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.394601    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-nhsd4_aa536561-ee97-4596-8730-24c79819ef86/prometheus-operator/0.log"
Jul 14 01:35:37.407835 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.407812    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-64785b6675-nhsd4_aa536561-ee97-4596-8730-24c79819ef86/kube-rbac-proxy/0.log"
Jul 14 01:35:37.428824 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:37.428777    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-77d98bc786-z7sp5_cf2da1d0-434f-4acd-9435-1958b73fc513/prometheus-operator-admission-webhook/0.log"
Jul 14 01:35:38.755348 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.755288    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"]
Jul 14 01:35:38.761167 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.761140    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.767059 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.767030    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"]
Jul 14 01:35:38.817121 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.817059    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-sys\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.817352 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.817147    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-proc\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.817352 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.817174    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-podres\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.817352 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.817230    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-lib-modules\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.817352 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.817300    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsksb\" (UniqueName: \"kubernetes.io/projected/5714474d-8535-46e2-b439-54e9050dd3da-kube-api-access-hsksb\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.918538 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918474    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-proc\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.918722 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-podres\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.918722 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918600    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-lib-modules\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.918722 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918667    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsksb\" (UniqueName: \"kubernetes.io/projected/5714474d-8535-46e2-b439-54e9050dd3da-kube-api-access-hsksb\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.918722 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918705    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-sys\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.918957 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918815    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-sys\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.918957 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918869    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-proc\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.919066 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.918968    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-podres\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.919066 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.919041    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5714474d-8535-46e2-b439-54e9050dd3da-lib-modules\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:38.927587 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:38.927546    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsksb\" (UniqueName: \"kubernetes.io/projected/5714474d-8535-46e2-b439-54e9050dd3da-kube-api-access-hsksb\") pod \"perf-node-gather-daemonset-jlnbm\" (UID: \"5714474d-8535-46e2-b439-54e9050dd3da\") " pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:39.081282 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:39.081241    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:39.271740 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:39.271302    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"]
Jul 14 01:35:40.293654 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:40.293612    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm" event={"ID":"5714474d-8535-46e2-b439-54e9050dd3da","Type":"ContainerStarted","Data":"d95e02157a31306572c44eee8e615ca92d69d884aaee94a95bfd19856f3568b1"}
Jul 14 01:35:40.294152 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:40.293688    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm" event={"ID":"5714474d-8535-46e2-b439-54e9050dd3da","Type":"ContainerStarted","Data":"f7005ab3d22f23cce72a4c2dfb068ae1fe05a56a69f4a497fab1b966dd51c10d"}
Jul 14 01:35:40.294488 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:40.294461    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:40.309547 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:40.309466    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm" podStartSLOduration=2.309443124 podStartE2EDuration="2.309443124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-14 01:35:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-14 01:35:40.307878804 +0000 UTC m=+3090.288963489" watchObservedRunningTime="2026-07-14 01:35:40.309443124 +0000 UTC m=+3090.290527787"
Jul 14 01:35:40.362055 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:40.362017    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-storage-operator_volume-data-source-validator-78495597f6-p5ltk_fd5d08e6-b26e-4d6b-b21d-f9d23ed05783/volume-data-source-validator/0.log"
Jul 14 01:35:41.406261 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:41.406231    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qqm2z_7cdd529a-053a-4f90-8e1f-88143f580ff9/dns/0.log"
Jul 14 01:35:41.428199 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:41.428172    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-qqm2z_7cdd529a-053a-4f90-8e1f-88143f580ff9/kube-rbac-proxy/0.log"
Jul 14 01:35:41.466457 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:41.466431    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gbfs8_44023eff-43a4-4016-a62c-f10e986d2a18/dns-node-resolver/0.log"
Jul 14 01:35:42.051791 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:42.051758    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-wq6ck_a13d6513-ea94-485d-adf6-532636f8112c/node-ca/0.log"
Jul 14 01:35:43.199357 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:43.199324    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f876ccc94-wrs6n_93bb0628-6a62-4442-a27e-bd8e614434b5/kube-auth-proxy/0.log"
Jul 14 01:35:44.056543 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:44.056493    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-lkgk6_3d478348-8510-4b2a-bb83-45abe05300d0/serve-healthcheck-canary/0.log"
Jul 14 01:35:44.596189 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:44.596153    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qsrmk_7629acf0-e2b5-406d-ba54-d1dca19b1bc9/kube-rbac-proxy/0.log"
Jul 14 01:35:44.614493 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:44.614466    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qsrmk_7629acf0-e2b5-406d-ba54-d1dca19b1bc9/exporter/0.log"
Jul 14 01:35:44.633989 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:44.633963    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-qsrmk_7629acf0-e2b5-406d-ba54-d1dca19b1bc9/extractor/0.log"
Jul 14 01:35:47.319343 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:47.319309    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-nwk5x/perf-node-gather-daemonset-jlnbm"
Jul 14 01:35:47.398351 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:47.398322    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7d6985f7-s4wp8_1eb3d285-aa44-4bf9-b85b-81ed74cb1be1/manager/0.log"
Jul 14 01:35:47.413071 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:47.413040    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-6b7d6985f7-s4wp8_1eb3d285-aa44-4bf9-b85b-81ed74cb1be1/copy-manifests/0.log"
Jul 14 01:35:48.640640 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:48.640609    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bd544bbf-pbkhm_8b92f0f2-2bd3-42b4-862b-bc17acaa08e1/manager/0.log"
Jul 14 01:35:49.611398 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:49.607657    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f876ccc94-wrs6n_93bb0628-6a62-4442-a27e-bd8e614434b5/kube-auth-proxy/0.log"
Jul 14 01:35:51.237014 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:51.236942    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-5f876ccc94-wrs6n_93bb0628-6a62-4442-a27e-bd8e614434b5/kube-auth-proxy/0.log"
Jul 14 01:35:53.734076 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:53.734048    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-b49lh_8e04aff9-8c81-4a11-9cd2-64c312934cff/migrator/0.log"
Jul 14 01:35:53.750578 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:53.750558    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-7f59b54f6c-b49lh_8e04aff9-8c81-4a11-9cd2-64c312934cff/graceful-termination/0.log"
Jul 14 01:35:54.992458 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:54.992428    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4szch_58c65e1b-6b8e-499e-8009-1b5ae97ecf47/kube-multus-additional-cni-plugins/0.log"
Jul 14 01:35:55.015426 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.015393    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4szch_58c65e1b-6b8e-499e-8009-1b5ae97ecf47/egress-router-binary-copy/0.log"
Jul 14 01:35:55.034464 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.034439    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4szch_58c65e1b-6b8e-499e-8009-1b5ae97ecf47/cni-plugins/0.log"
Jul 14 01:35:55.050802 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.050771    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4szch_58c65e1b-6b8e-499e-8009-1b5ae97ecf47/bond-cni-plugin/0.log"
Jul 14 01:35:55.066459 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.066434    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4szch_58c65e1b-6b8e-499e-8009-1b5ae97ecf47/routeoverride-cni/0.log"
Jul 14 01:35:55.084613 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.084595    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4szch_58c65e1b-6b8e-499e-8009-1b5ae97ecf47/whereabouts-cni-bincopy/0.log"
Jul 14 01:35:55.101659 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.101623    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-4szch_58c65e1b-6b8e-499e-8009-1b5ae97ecf47/whereabouts-cni/0.log"
Jul 14 01:35:55.431278 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.431249    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-jjgh9_eebeab35-9893-4fb3-b2e0-c5f66e9e520e/kube-multus/0.log"
Jul 14 01:35:55.488174 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.488149    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7bcw7_79b4894c-cac8-408b-ae28-e3c6788ba884/network-metrics-daemon/0.log"
Jul 14 01:35:55.502946 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:55.502922    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-7bcw7_79b4894c-cac8-408b-ae28-e3c6788ba884/kube-rbac-proxy/0.log"
Jul 14 01:35:56.580745 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.580696    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-controller/0.log"
Jul 14 01:35:56.594978 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.594946    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/0.log"
Jul 14 01:35:56.594978 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.594985    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovn-acl-logging/1.log"
Jul 14 01:35:56.617376 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.617342    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/kube-rbac-proxy-node/0.log"
Jul 14 01:35:56.640771 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.640742    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/kube-rbac-proxy-ovn-metrics/0.log"
Jul 14 01:35:56.655621 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.655592    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/northd/0.log"
Jul 14 01:35:56.673118 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.673091    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/nbdb/0.log"
Jul 14 01:35:56.691148 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.691121    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/sbdb/0.log"
Jul 14 01:35:56.774292 ip-10-0-129-125 kubenswrapper[2552]: I0714 01:35:56.774215    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-k45vj_14cac740-f5b9-4a25-bcb2-3d12f82f8fce/ovnkube-controller/0.log"