Jul 24 20:13:18.774301 ip-10-0-129-245 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Jul 24 20:13:18.774310 ip-10-0-129-245 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Jul 24 20:13:18.774317 ip-10-0-129-245 systemd[1]: kubelet.service: Failed with result 'resources'.
Jul 24 20:13:18.774565 ip-10-0-129-245 systemd[1]: Failed to start Kubernetes Kubelet.
Jul 24 20:13:30.274270 ip-10-0-129-245 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Jul 24 20:13:30.274288 ip-10-0-129-245 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7b3f513d1564473ca9499b401cfc4093 --
Jul 24 20:15:24.962805 ip-10-0-129-245 systemd[1]: Starting Kubernetes Kubelet...
Jul 24 20:15:25.342006 ip-10-0-129-245 kubenswrapper[2566]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 20:15:25.342006 ip-10-0-129-245 kubenswrapper[2566]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Jul 24 20:15:25.342006 ip-10-0-129-245 kubenswrapper[2566]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 20:15:25.342006 ip-10-0-129-245 kubenswrapper[2566]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Jul 24 20:15:25.342006 ip-10-0-129-245 kubenswrapper[2566]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Jul 24 20:15:25.342722 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.342243    2566 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Jul 24 20:15:25.346938 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346922    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 20:15:25.346938 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346937    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346941    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346945    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346948    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346951    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346954    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346957    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346959    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346962    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346965    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346968    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346970    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346973    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346975    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346978    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346980    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346983    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346985    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346988    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346990    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 20:15:25.347000 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346993    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346995    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.346998    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347000    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347003    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347006    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347009    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347019    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347022    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347024    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347027    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347029    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347032    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347034    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347037    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347039    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347042    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347044    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347046    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 20:15:25.347491 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347049    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347052    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347054    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347057    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347059    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347061    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347064    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347066    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347069    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347072    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347075    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347077    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347080    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347082    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347084    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347088    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347091    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347093    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347096    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347098    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 20:15:25.347963 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347101    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347103    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347106    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347108    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347111    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347113    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347117    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347119    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347123    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347127    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347130    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347132    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347135    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347137    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347140    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347142    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347145    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347148    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347152    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347155    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 20:15:25.348438 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347159    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347162    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347165    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347168    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347171    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347173    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347176    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347179    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347184    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347187    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347190    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347193    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347196    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347199    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347202    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347205    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347208    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347210    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.347213    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348600    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 20:15:25.348932 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348608    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348611    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348614    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348617    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348620    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348623    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348626    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348629    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348632    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348634    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348637    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348640    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348642    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348646    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348649    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348652    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348654    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348657    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348660    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348663    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 20:15:25.349405 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348666    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348668    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348671    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348674    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348676    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348679    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348681    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348684    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348686    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348689    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348691    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348694    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348696    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348699    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348701    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348704    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348707    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348709    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348712    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348714    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 20:15:25.349894 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348716    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348719    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348723    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348725    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348729    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348733    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348736    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348739    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348741    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348744    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348747    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348752    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348755    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348758    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348761    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348763    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348766    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348768    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348771    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 20:15:25.350375 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348773    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348776    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348778    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348781    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348783    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348786    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348788    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348796    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348801    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348803    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348806    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348809    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348811    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348814    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348816    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348819    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348821    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348824    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348827    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 20:15:25.350840 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348829    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348834    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348837    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348840    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348843    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348846    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348850    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348853    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348856    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348858    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348860    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348863    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348868    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348871    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348873    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348876    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348878    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348881    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348883    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.348886    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 20:15:25.351291 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349256    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349262    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349264    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349267    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349269    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349272    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349275    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349277    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349280    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349283    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349285    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349288    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349290    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349293    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349296    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349299    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349301    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349304    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349306    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349309    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 20:15:25.351789 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349311    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349313    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349316    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349318    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349322    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349325    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349327    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349330    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349332    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349335    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349337    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349340    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349342    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349344    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349347    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349349    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349352    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349361    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349364    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 20:15:25.352265 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349367    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349370    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349372    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349375    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349377    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349380    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349382    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349385    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349388    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349391    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349393    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349396    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349398    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349401    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349403    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349405    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349408    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349412    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349415    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349417    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 20:15:25.352765 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349420    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349423    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349425    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349427    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349430    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349432    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349435    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349437    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349440    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349444    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349447    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349450    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349453    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349456    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349459    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349461    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349464    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349467    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349469    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349471    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 20:15:25.353244 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349474    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349479    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349482    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349485    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349488    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349490    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349493    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349495    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349497    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349500    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349503    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349505    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349508    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349510    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349513    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349515    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349518    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349520    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349536    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 20:15:25.353776 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.349539    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349606    2566 flags.go:64] FLAG: --address="0.0.0.0"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349613    2566 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349619    2566 flags.go:64] FLAG: --anonymous-auth="true"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349623    2566 flags.go:64] FLAG: --application-metrics-count-limit="100"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349627    2566 flags.go:64] FLAG: --authentication-token-webhook="false"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349630    2566 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349634    2566 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349638    2566 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349641    2566 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349644    2566 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349648    2566 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349651    2566 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349654    2566 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349660    2566 flags.go:64] FLAG: --cgroup-root=""
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349663    2566 flags.go:64] FLAG: --cgroups-per-qos="true"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349666    2566 flags.go:64] FLAG: --client-ca-file=""
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349669    2566 flags.go:64] FLAG: --cloud-provider="external"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349672    2566 flags.go:64] FLAG: --cluster-dns="[]"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349677    2566 flags.go:64] FLAG: --cluster-domain=""
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349679    2566 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349682    2566 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349686    2566 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349689    2566 flags.go:64] FLAG: --container-log-max-files="5"
Jul 24 20:15:25.354222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349693    2566 flags.go:64] FLAG: --container-log-max-size="10Mi"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349696    2566 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349699    2566 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349702    2566 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349705    2566 flags.go:64] FLAG: --contention-profiling="false"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349708    2566 flags.go:64] FLAG: --cpu-cfs-quota="true"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349711    2566 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349714    2566 flags.go:64] FLAG: --cpu-manager-policy="none"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349717    2566 flags.go:64] FLAG: --cpu-manager-policy-options=""
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349721    2566 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349724    2566 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349726    2566 flags.go:64] FLAG: --enable-debugging-handlers="true"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349729    2566 flags.go:64] FLAG: --enable-load-reader="false"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349732    2566 flags.go:64] FLAG: --enable-server="true"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349735    2566 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349739    2566 flags.go:64] FLAG: --event-burst="100"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349742    2566 flags.go:64] FLAG: --event-qps="50"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349745    2566 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349748    2566 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349750    2566 flags.go:64] FLAG: --eviction-hard=""
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349754    2566 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349757    2566 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349762    2566 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349766    2566 flags.go:64] FLAG: --eviction-soft=""
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349769    2566 flags.go:64] FLAG: --eviction-soft-grace-period=""
Jul 24 20:15:25.354799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349772    2566 flags.go:64] FLAG: --exit-on-lock-contention="false"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349775    2566 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349778    2566 flags.go:64] FLAG: --experimental-mounter-path=""
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349781    2566 flags.go:64] FLAG: --fail-cgroupv1="false"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349784    2566 flags.go:64] FLAG: --fail-swap-on="true"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349787    2566 flags.go:64] FLAG: --feature-gates=""
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349790    2566 flags.go:64] FLAG: --file-check-frequency="20s"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349793    2566 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349796    2566 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349799    2566 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349802    2566 flags.go:64] FLAG: --healthz-port="10248"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349805    2566 flags.go:64] FLAG: --help="false"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349807    2566 flags.go:64] FLAG: --hostname-override="ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349810    2566 flags.go:64] FLAG: --housekeeping-interval="10s"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349813    2566 flags.go:64] FLAG: --http-check-frequency="20s"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349816    2566 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349819    2566 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349822    2566 flags.go:64] FLAG: --image-gc-high-threshold="85"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349825    2566 flags.go:64] FLAG: --image-gc-low-threshold="80"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349828    2566 flags.go:64] FLAG: --image-service-endpoint=""
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349830    2566 flags.go:64] FLAG: --kernel-memcg-notification="false"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349833    2566 flags.go:64] FLAG: --kube-api-burst="100"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349836    2566 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349839    2566 flags.go:64] FLAG: --kube-api-qps="50"
Jul 24 20:15:25.355424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349841    2566 flags.go:64] FLAG: --kube-reserved=""
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349844    2566 flags.go:64] FLAG: --kube-reserved-cgroup=""
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349847    2566 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349850    2566 flags.go:64] FLAG: --kubelet-cgroups=""
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349852    2566 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349856    2566 flags.go:64] FLAG: --lock-file=""
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349859    2566 flags.go:64] FLAG: --log-cadvisor-usage="false"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349863    2566 flags.go:64] FLAG: --log-flush-frequency="5s"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349866    2566 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349871    2566 flags.go:64] FLAG: --log-json-split-stream="false"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349874    2566 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349877    2566 flags.go:64] FLAG: --log-text-split-stream="false"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349879    2566 flags.go:64] FLAG: --logging-format="text"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349882    2566 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349885    2566 flags.go:64] FLAG: --make-iptables-util-chains="true"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349888    2566 flags.go:64] FLAG: --manifest-url=""
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349891    2566 flags.go:64] FLAG: --manifest-url-header=""
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349896    2566 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349899    2566 flags.go:64] FLAG: --max-open-files="1000000"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349903    2566 flags.go:64] FLAG: --max-pods="110"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349906    2566 flags.go:64] FLAG: --maximum-dead-containers="-1"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349909    2566 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349912    2566 flags.go:64] FLAG: --memory-manager-policy="None"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349914    2566 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349917    2566 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Jul 24 20:15:25.356040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349920    2566 flags.go:64] FLAG: --node-ip="0.0.0.0"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349924    2566 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349930    2566 flags.go:64] FLAG: --node-status-max-images="50"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349933    2566 flags.go:64] FLAG: --node-status-update-frequency="10s"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349936    2566 flags.go:64] FLAG: --oom-score-adj="-999"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349941    2566 flags.go:64] FLAG: --pod-cidr=""
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349944    2566 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:7505336a406394b8c1c77d14d890a9f628aa4cfff172312179cb9af8c28c30bb"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349949    2566 flags.go:64] FLAG: --pod-manifest-path=""
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349952    2566 flags.go:64] FLAG: --pod-max-pids="-1"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349955    2566 flags.go:64] FLAG: --pods-per-core="0"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349958    2566 flags.go:64] FLAG: --port="10250"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349961    2566 flags.go:64] FLAG: --protect-kernel-defaults="false"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349964    2566 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-08dd5cb07b18ed0a7"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349968    2566 flags.go:64] FLAG: --qos-reserved=""
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349971    2566 flags.go:64] FLAG: --read-only-port="10255"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349975    2566 flags.go:64] FLAG: --register-node="true"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349978    2566 flags.go:64] FLAG: --register-with-taints=""
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349981    2566 flags.go:64] FLAG: --registry-burst="10"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349984    2566 flags.go:64] FLAG: --registry-qps="5"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349987    2566 flags.go:64] FLAG: --reserved-cpus=""
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349989    2566 flags.go:64] FLAG: --reserved-memory=""
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349993    2566 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349996    2566 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.349998    2566 flags.go:64] FLAG: --rotate-certificates="false"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350001    2566 flags.go:64] FLAG: --rotate-server-certificates="false"
Jul 24 20:15:25.356657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350004    2566 flags.go:64] FLAG: --runonce="false"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350007    2566 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350010    2566 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350012    2566 flags.go:64] FLAG: --seccomp-default="false"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350015    2566 flags.go:64] FLAG: --serialize-image-pulls="true"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350018    2566 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350021    2566 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350025    2566 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350027    2566 flags.go:64] FLAG: --storage-driver-password="root"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350030    2566 flags.go:64] FLAG: --storage-driver-secure="false"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350033    2566 flags.go:64] FLAG: --storage-driver-table="stats"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350035    2566 flags.go:64] FLAG: --storage-driver-user="root"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350039    2566 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350042    2566 flags.go:64] FLAG: --sync-frequency="1m0s"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350045    2566 flags.go:64] FLAG: --system-cgroups=""
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350048    2566 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350053    2566 flags.go:64] FLAG: --system-reserved-cgroup=""
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350056    2566 flags.go:64] FLAG: --tls-cert-file=""
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350059    2566 flags.go:64] FLAG: --tls-cipher-suites="[]"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350063    2566 flags.go:64] FLAG: --tls-min-version=""
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350067    2566 flags.go:64] FLAG: --tls-private-key-file=""
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350070    2566 flags.go:64] FLAG: --topology-manager-policy="none"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350072    2566 flags.go:64] FLAG: --topology-manager-policy-options=""
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350076    2566 flags.go:64] FLAG: --topology-manager-scope="container"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350079    2566 flags.go:64] FLAG: --v="2"
Jul 24 20:15:25.357260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350083    2566 flags.go:64] FLAG: --version="false"
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350086    2566 flags.go:64] FLAG: --vmodule=""
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350090    2566 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.350093    2566 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350201    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350206    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350209    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350212    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350214    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350217    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350220    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350223    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350226    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350229    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350231    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350234    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350236    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350239    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350242    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 20:15:25.357876 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350247    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350250    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350253    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350256    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350258    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350261    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350263    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350266    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350270    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350273    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350276    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350278    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350281    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350283    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350286    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350288    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350291    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350293    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350296    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350298    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 20:15:25.358326 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350301    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350303    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350306    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350308    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350310    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350313    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350315    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350318    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350320    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350323    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350326    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350328    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350333    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350335    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350339    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350341    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350343    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350346    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350348    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350351    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 20:15:25.358850 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350356    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350359    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350361    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350365    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350369    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350372    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350376    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350379    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350381    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350384    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350387    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350390    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350392    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350395    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350397    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350400    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350402    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350405    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350407    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350410    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 20:15:25.359318 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350412    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350415    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350418    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350420    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350424    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350426    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350428    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350431    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350433    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350436    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350438    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350441    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350445    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350447    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350450    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350452    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350455    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350457    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350460    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350463    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 20:15:25.359873 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350465    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350468    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350470    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.350473    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.351121    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.357623    2566 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.357637    2566 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.357657    2566 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.357670    2566 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357726    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357730    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357734    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357739    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357743    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357748    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 20:15:25.360338 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357751    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357754    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357757    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357760    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357762    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357765    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357768    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357770    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357773    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357775    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357778    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357781    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357783    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357786    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357788    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357791    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357795    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357798    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357801    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 20:15:25.360702 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357804    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357807    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357810    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357814    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357820    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357825    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357829    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357833    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357836    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357838    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357841    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357843    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357847    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357849    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357852    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357854    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357857    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357859    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357862    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 20:15:25.361165 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357864    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357867    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357869    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357872    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357874    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357877    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357879    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357882    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357885    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357888    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357892    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357897    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357901    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357904    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357907    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357909    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357912    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357915    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357918    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357920    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 20:15:25.361634 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357923    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357925    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357928    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357931    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357934    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357937    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357940    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357943    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357946    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357948    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357951    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357954    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357957    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357959    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357962    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357964    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357969    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357972    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357976    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357980    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 20:15:25.362140 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357983    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357986    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357988    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357991    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357993    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357996    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.357999    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358002    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358004    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358007    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358010    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358013    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358015    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358018    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358020    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 20:15:25.362622 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.358025    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358162    2566 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358167    2566 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358170    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358173    2566 feature_gate.go:328] unrecognized feature gate: NewOLM
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358176    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358178    2566 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358181    2566 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358184    2566 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358187    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358189    2566 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358192    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358194    2566 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358198    2566 feature_gate.go:328] unrecognized feature gate: EventTTL
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358202    2566 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358206    2566 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358210    2566 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358213    2566 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358216    2566 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358218    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Jul 24 20:15:25.362976 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358221    2566 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358223    2566 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358225    2566 feature_gate.go:328] unrecognized feature gate: OSStreams
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358229    2566 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358231    2566 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358234    2566 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358236    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358239    2566 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358241    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358244    2566 feature_gate.go:328] unrecognized feature gate: Example2
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358247    2566 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358249    2566 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358252    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358254    2566 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358257    2566 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358259    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358261    2566 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358264    2566 feature_gate.go:328] unrecognized feature gate: DualReplica
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358266    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358269    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Jul 24 20:15:25.363466 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358271    2566 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358274    2566 feature_gate.go:328] unrecognized feature gate: PinnedImages
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358279    2566 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358283    2566 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358286    2566 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358289    2566 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358291    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358293    2566 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358297    2566 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358302    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358304    2566 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358307    2566 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358310    2566 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358314    2566 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358316    2566 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358319    2566 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358322    2566 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358324    2566 feature_gate.go:328] unrecognized feature gate: OVNObservability
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358327    2566 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Jul 24 20:15:25.363978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358330    2566 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358333    2566 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358335    2566 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358339    2566 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358342    2566 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358344    2566 feature_gate.go:328] unrecognized feature gate: Example
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358347    2566 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358350    2566 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358354    2566 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358358    2566 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358362    2566 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358365    2566 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358367    2566 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358370    2566 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358372    2566 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358375    2566 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358377    2566 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358379    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358382    2566 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Jul 24 20:15:25.364425 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358384    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358387    2566 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358390    2566 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358393    2566 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358396    2566 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358399    2566 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358401    2566 feature_gate.go:328] unrecognized feature gate: SignatureStores
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358404    2566 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358406    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358409    2566 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358411    2566 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358414    2566 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358416    2566 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358419    2566 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358421    2566 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358424    2566 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358427    2566 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358432    2566 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358436    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Jul 24 20:15:25.364978 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358440    2566 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358443    2566 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:25.358445    2566 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.358450    2566 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.358569    2566 server.go:962] "Client rotation is on, will bootstrap in background"
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.360540    2566 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.361686    2566 server.go:1019] "Starting client certificate rotation"
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.361783    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.362192    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.363884    2566 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 20:15:25.365435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.364996    2566 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Jul 24 20:15:25.374021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.373999    2566 log.go:25] "Validated CRI v1 runtime API"
Jul 24 20:15:25.376408 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.376389    2566 log.go:25] "Validated CRI v1 image API"
Jul 24 20:15:25.377936 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.377921    2566 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Jul 24 20:15:25.381635 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.381613    2566 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 ccd2584b-1795-40fe-82a9-b168eca5be79:/dev/nvme0n1p3 e5721284-9d7a-43ec-a60c-3d2a57e49ce8:/dev/nvme0n1p4]
Jul 24 20:15:25.381696 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.381635    2566 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Jul 24 20:15:25.387229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.387119    2566 manager.go:216] Machine: {Timestamp:2026-07-24 20:15:25.385937307 +0000 UTC m=+0.312076221 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3174861 MemoryCapacity:33164496896 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec21f665ea08f1e1ab6041786bfb5ae7 SystemUUID:ec21f665-ea08-f1e1-ab60-41786bfb5ae7 BootID:7b3f513d-1564-473c-a949-9b401cfc4093 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582250496 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582246400 Type:vfs Inodes:4048400 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632902656 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:e0:f9:01:0d:d7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:e0:f9:01:0d:d7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:96:7c:c9:b3:5d:1f Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164496896 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Jul 24 20:15:25.387229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.387222    2566 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Jul 24 20:15:25.387350 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.387302    2566 manager.go:232] Version: {KernelVersion:5.14.0-570.129.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260717-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Jul 24 20:15:25.390166 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.390137    2566 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Jul 24 20:15:25.390296 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.390167    2566 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-245.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Jul 24 20:15:25.390881 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.390870    2566 topology_manager.go:138] "Creating topology manager with none policy"
Jul 24 20:15:25.390920 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.390883    2566 container_manager_linux.go:309] "Creating device plugin manager"
Jul 24 20:15:25.390920 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.390890    2566 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 20:15:25.391119 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.391103    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 20:15:25.391548 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.391538    2566 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Jul 24 20:15:25.392349 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.392340    2566 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Jul 24 20:15:25.393803 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.393791    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 20:15:25.393920 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.393910    2566 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Jul 24 20:15:25.393959 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.393942    2566 kubelet.go:486] "Attempting to sync node with API server"
Jul 24 20:15:25.393959 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.393954    2566 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Jul 24 20:15:25.394009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.393969    2566 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Jul 24 20:15:25.394009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.393979    2566 kubelet.go:398] "Adding apiserver pod source"
Jul 24 20:15:25.394009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.393991    2566 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Jul 24 20:15:25.394372 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.394360    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 20:15:25.394421 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.394377    2566 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Jul 24 20:15:25.396981 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.396958    2566 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.10-2.rhaos4.21.git83ee25b.el9" apiVersion="v1"
Jul 24 20:15:25.398355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.398338    2566 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Jul 24 20:15:25.398417 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.398370    2566 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Jul 24 20:15:25.400060 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400049    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Jul 24 20:15:25.400098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400066    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Jul 24 20:15:25.400098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400073    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Jul 24 20:15:25.400098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400079    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Jul 24 20:15:25.400098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400087    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Jul 24 20:15:25.400098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400097    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Jul 24 20:15:25.400227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400109    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Jul 24 20:15:25.400227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400115    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Jul 24 20:15:25.400227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400122    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Jul 24 20:15:25.400227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400127    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Jul 24 20:15:25.400227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400140    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Jul 24 20:15:25.400227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400146    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Jul 24 20:15:25.400847 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400834    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Jul 24 20:15:25.400908 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.400848    2566 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Jul 24 20:15:25.402855 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.402823    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Jul 24 20:15:25.403140 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.403114    2566 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-245.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Jul 24 20:15:25.403197 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.403129    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Jul 24 20:15:25.404339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.404327    2566 server.go:1268] "Started kubelet"
Jul 24 20:15:25.404445 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.404404    2566 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Jul 24 20:15:25.404514 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.404445    2566 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Jul 24 20:15:25.404514 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.404468    2566 server_v1.go:49] "podresources" method="list" useActivePods=true
Jul 24 20:15:25.405347 ip-10-0-129-245 systemd[1]: Started Kubernetes Kubelet.
Jul 24 20:15:25.406314 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.406221    2566 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Jul 24 20:15:25.406314 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.406307    2566 server.go:310] "Adding debug handlers to kubelet server"
Jul 24 20:15:25.410632 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.410611    2566 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Jul 24 20:15:25.410863 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.410846    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z2jwn"
Jul 24 20:15:25.413234 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.413218    2566 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Jul 24 20:15:25.413234 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.413215    2566 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Jul 24 20:15:25.413924 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.413906    2566 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Jul 24 20:15:25.414037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.413829    2566 volume_manager.go:311] "The desired_state_of_world populator starts"
Jul 24 20:15:25.414085 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.414044    2566 volume_manager.go:313] "Starting Kubelet Volume Manager"
Jul 24 20:15:25.414327 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.414310    2566 reconstruct.go:97] "Volume reconstruction finished"
Jul 24 20:15:25.414472 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.414460    2566 reconciler.go:29] "Reconciler: start to sync state"
Jul 24 20:15:25.414538 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.414309    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:25.415643 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.415618    2566 factory.go:55] Registering systemd factory
Jul 24 20:15:25.415726 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.415651    2566 factory.go:223] Registration of the systemd container factory successfully
Jul 24 20:15:25.415984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.415964    2566 factory.go:153] Registering CRI-O factory
Jul 24 20:15:25.415984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.415982    2566 factory.go:223] Registration of the crio container factory successfully
Jul 24 20:15:25.416119 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.416031    2566 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Jul 24 20:15:25.416119 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.416060    2566 factory.go:103] Registering Raw factory
Jul 24 20:15:25.416119 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.416075    2566 manager.go:1196] Started watching for new ooms in manager
Jul 24 20:15:25.416568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.416554    2566 manager.go:349] Starting recovery of all containers
Jul 24 20:15:25.418815 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.418788    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-z2jwn"
Jul 24 20:15:25.418904 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.418887    2566 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Jul 24 20:15:25.419027 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.419007    2566 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-245.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Jul 24 20:15:25.419832 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.418994    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-245.ec2.internal.18c55308fdbf8b6f  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-245.ec2.internal,UID:ip-10-0-129-245.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-245.ec2.internal,},FirstTimestamp:2026-07-24 20:15:25.404302191 +0000 UTC m=+0.330441086,LastTimestamp:2026-07-24 20:15:25.404302191 +0000 UTC m=+0.330441086,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-245.ec2.internal,}"
Jul 24 20:15:25.428780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.428630    2566 manager.go:354] Recovery completed
Jul 24 20:15:25.432654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.432640    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 20:15:25.434424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.434404    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 20:15:25.434541 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.434439    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 20:15:25.434541 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.434453    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 24 20:15:25.435001 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.434986    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 20:15:25.435069 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.434989    2566 cpu_manager.go:222] "Starting CPU manager" policy="none"
Jul 24 20:15:25.435069 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.435046    2566 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Jul 24 20:15:25.435069 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.435063    2566 state_mem.go:36] "Initialized new in-memory state store"
Jul 24 20:15:25.435625 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.435610    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 20:15:25.435678 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.435662    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 20:15:25.435678 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.435675    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 24 20:15:25.436482 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.436472    2566 policy_none.go:49] "None policy: Start"
Jul 24 20:15:25.436482 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.436416    2566 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-245.ec2.internal.18c55308ff8b26e4  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-245.ec2.internal,UID:ip-10-0-129-245.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-245.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-245.ec2.internal,},FirstTimestamp:2026-07-24 20:15:25.434423012 +0000 UTC m=+0.360561917,LastTimestamp:2026-07-24 20:15:25.434423012 +0000 UTC m=+0.360561917,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-245.ec2.internal,}"
Jul 24 20:15:25.436604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.436486    2566 memory_manager.go:187] "Starting memorymanager" policy="None"
Jul 24 20:15:25.436604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.436495    2566 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Jul 24 20:15:25.437451 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.437435    2566 policy_none.go:47] "Start"
Jul 24 20:15:25.469097 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.469073    2566 manager.go:337] "Starting Device Plugin manager"
Jul 24 20:15:25.469195 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.469110    2566 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Jul 24 20:15:25.469195 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.469123    2566 server.go:87] "Starting device plugin registration server"
Jul 24 20:15:25.469404 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.469391    2566 eviction_manager.go:189] "Eviction manager: starting control loop"
Jul 24 20:15:25.469459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.469407    2566 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Jul 24 20:15:25.469561 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.469544    2566 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Jul 24 20:15:25.469656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.469637    2566 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Jul 24 20:15:25.469656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.469651    2566 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Jul 24 20:15:25.470202 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.470182    2566 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Jul 24 20:15:25.470275 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.470225    2566 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:25.571999 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.571962    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 20:15:25.573221 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.573195    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 20:15:25.573308 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.573225    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 20:15:25.573308 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.573239    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 24 20:15:25.573308 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.573272    2566 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.582169 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.582151    2566 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.582228 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.582180    2566 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-245.ec2.internal\": node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:25.599602 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.599528    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:25.684756 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.684714    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Jul 24 20:15:25.685950 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.685929    2566 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Jul 24 20:15:25.686031 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.685982    2566 status_manager.go:244] "Starting to sync pod status with apiserver"
Jul 24 20:15:25.686031 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.686013    2566 kubelet.go:2448] "Starting kubelet main sync loop"
Jul 24 20:15:25.686131 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.686060    2566 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Jul 24 20:15:25.688381 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.688363    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 20:15:25.699828 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.699805    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:25.788490 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.788436    2566 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"]
Jul 24 20:15:25.788651 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.788565    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 20:15:25.789689 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.789663    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 20:15:25.789807 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.789702    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 20:15:25.789807 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.789714    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 24 20:15:25.791300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.791287    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 20:15:25.791489 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.791474    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.791529 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.791507    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 20:15:25.792049 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.792032    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 20:15:25.792114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.792069    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 20:15:25.792114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.792081    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 24 20:15:25.792114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.792070    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 20:15:25.792114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.792108    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 20:15:25.792264 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.792123    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 24 20:15:25.793277 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.793262    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.793338 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.793290    2566 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Jul 24 20:15:25.793979 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.793963    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientMemory"
Jul 24 20:15:25.794049 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.793989    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasNoDiskPressure"
Jul 24 20:15:25.794049 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.794003    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeHasSufficientPID"
Jul 24 20:15:25.800682 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.800665    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:25.815692 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.815666    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0b7853205bd7e1e30de35767acd8e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"2e0b7853205bd7e1e30de35767acd8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.815789 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.815699    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0b7853205bd7e1e30de35767acd8e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"2e0b7853205bd7e1e30de35767acd8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.815789 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.815716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58be0bbdb2076ba8a2b4b2e2c5986370-config\") pod \"kube-apiserver-proxy-ip-10-0-129-245.ec2.internal\" (UID: \"58be0bbdb2076ba8a2b4b2e2c5986370\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.825561 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.825543    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-245.ec2.internal\" not found" node="ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.830263 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.830245    2566 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-245.ec2.internal\" not found" node="ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.900920 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:25.900819    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:25.916217 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.916191    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0b7853205bd7e1e30de35767acd8e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"2e0b7853205bd7e1e30de35767acd8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.916315 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.916227    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58be0bbdb2076ba8a2b4b2e2c5986370-config\") pod \"kube-apiserver-proxy-ip-10-0-129-245.ec2.internal\" (UID: \"58be0bbdb2076ba8a2b4b2e2c5986370\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.916315 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.916251    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0b7853205bd7e1e30de35767acd8e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"2e0b7853205bd7e1e30de35767acd8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.916315 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.916302    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2e0b7853205bd7e1e30de35767acd8e1-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"2e0b7853205bd7e1e30de35767acd8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.916439 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.916316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/58be0bbdb2076ba8a2b4b2e2c5986370-config\") pod \"kube-apiserver-proxy-ip-10-0-129-245.ec2.internal\" (UID: \"58be0bbdb2076ba8a2b4b2e2c5986370\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:25.916439 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:25.916305    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2e0b7853205bd7e1e30de35767acd8e1-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal\" (UID: \"2e0b7853205bd7e1e30de35767acd8e1\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:26.007351 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:26.007299    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:26.115048 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:26.115008    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:26.128291 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.128266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:26.132712 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.132579    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:26.215547 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:26.215436    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:26.320201 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:26.320161    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:26.345112 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.345087    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 20:15:26.361112 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.361091    2566 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Jul 24 20:15:26.361239 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.361221    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 20:15:26.361281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.361253    2566 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Jul 24 20:15:26.420602 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:26.420569    2566 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-245.ec2.internal\" not found"
Jul 24 20:15:26.423019 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.422998    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-07-23 20:10:26 +0000 UTC" deadline="2028-03-03 00:15:50.047601773 +0000 UTC"
Jul 24 20:15:26.423077 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.423020    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14092h0m23.624585035s"
Jul 24 20:15:26.430745 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.430729    2566 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 20:15:26.515758 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.515662    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:26.527696 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.527656    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 20:15:26.529538 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.529520    2566 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal"
Jul 24 20:15:26.538499 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.538480    2566 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Jul 24 20:15:26.670157 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:26.670124    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58be0bbdb2076ba8a2b4b2e2c5986370.slice/crio-d7303eb7ace79e2bba1e72184e2c861c630d927e4b254c691c05220edbe45e84 WatchSource:0}: Error finding container d7303eb7ace79e2bba1e72184e2c861c630d927e4b254c691c05220edbe45e84: Status 404 returned error can't find the container with id d7303eb7ace79e2bba1e72184e2c861c630d927e4b254c691c05220edbe45e84
Jul 24 20:15:26.670714 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:26.670686    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e0b7853205bd7e1e30de35767acd8e1.slice/crio-4859ca17aa50410bfa55ada6770220a816130d66717c0d7a4d4a0fd85105f019 WatchSource:0}: Error finding container 4859ca17aa50410bfa55ada6770220a816130d66717c0d7a4d4a0fd85105f019: Status 404 returned error can't find the container with id 4859ca17aa50410bfa55ada6770220a816130d66717c0d7a4d4a0fd85105f019
Jul 24 20:15:26.676699 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.675990    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 20:15:26.690192 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.690145    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal" event={"ID":"58be0bbdb2076ba8a2b4b2e2c5986370","Type":"ContainerStarted","Data":"d7303eb7ace79e2bba1e72184e2c861c630d927e4b254c691c05220edbe45e84"}
Jul 24 20:15:26.691280 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.691256    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" event={"ID":"2e0b7853205bd7e1e30de35767acd8e1","Type":"ContainerStarted","Data":"4859ca17aa50410bfa55ada6770220a816130d66717c0d7a4d4a0fd85105f019"}
Jul 24 20:15:26.709026 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:26.708997    2566 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 20:15:27.397190 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.397155    2566 apiserver.go:52] "Watching apiserver"
Jul 24 20:15:27.406882 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.406856    2566 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Jul 24 20:15:27.407323 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.407285    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-qh4xx","kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg","openshift-image-registry/node-ca-qt7hz","openshift-network-diagnostics/network-check-target-bls72","openshift-network-operator/iptables-alerter-zpxbt","openshift-ovn-kubernetes/ovnkube-node-5txqj","kube-system/global-pull-secret-syncer-gqxld","kube-system/konnectivity-agent-wd5fk","openshift-cluster-node-tuning-operator/tuned-lvk7m","openshift-dns/node-resolver-6gllf","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal","openshift-multus/multus-9kv2f","openshift-multus/multus-additional-cni-plugins-2pz67"]
Jul 24 20:15:27.409226 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.408967    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.410466 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.410441    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.411884 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.411675    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Jul 24 20:15:27.412310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.412063    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Jul 24 20:15:27.412310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.412115    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.412310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.412156    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-mv49b\""
Jul 24 20:15:27.413435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.413413    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.413778 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.413752    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Jul 24 20:15:27.414301 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.414263    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Jul 24 20:15:27.414547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.414525    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Jul 24 20:15:27.415732 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.415710    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Jul 24 20:15:27.415819 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.415776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Jul 24 20:15:27.416145 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.416123    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Jul 24 20:15:27.416497 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.416479    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Jul 24 20:15:27.416912 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.416892    2566 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Jul 24 20:15:27.416995 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.416946    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Jul 24 20:15:27.416995 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.416964    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Jul 24 20:15:27.417532 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.416478    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Jul 24 20:15:27.417532 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.417374    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5kvjf\""
Jul 24 20:15:27.417760 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.416905    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-cc9lj\""
Jul 24 20:15:27.417885 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.417866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:27.419113 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.419090    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Jul 24 20:15:27.419414 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.419364    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:27.421155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.421113    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.421285 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.421266    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Jul 24 20:15:27.421539 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.421519    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Jul 24 20:15:27.421933 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.421874    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Jul 24 20:15:27.422100 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.422081    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-kxddx\""
Jul 24 20:15:27.422865 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.422828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.423878 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.423851    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Jul 24 20:15:27.424001 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.423933    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Jul 24 20:15:27.424167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.424141    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-462vt\""
Jul 24 20:15:27.424268 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.424219    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Jul 24 20:15:27.424393 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.424375    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:27.424498 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.424475    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:27.425495 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.425471    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Jul 24 20:15:27.425604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.425554    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Jul 24 20:15:27.425724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.425709    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-qr2hb\""
Jul 24 20:15:27.425965 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.425949    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.427376 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427356    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovnkube-script-lib\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.427462 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-cni-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.427462 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427412    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.427592 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-socket-dir-parent\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.427652 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427590    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-socket-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.427652 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427616    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-device-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.427771 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9659\" (UniqueName: \"kubernetes.io/projected/68c5be09-ca2b-4a6c-a273-85c70f8650dc-kube-api-access-j9659\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.427771 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4dcaef25-bb54-46c5-8973-8d4dd38deb93-serviceca\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.427771 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427753    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:27.427941 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427780    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-slash\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.427998 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427930    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-systemd\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.427998 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.427988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-systemd-units\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-var-lib-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-cni-netd\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428107    2566 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 20:15:27.428253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428117    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovnkube-config\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428140    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-os-release\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.428253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428195    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-netns\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.428253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428233    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-cni-multus\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.428253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428260    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-registration-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428284    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-ovn\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428317    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-log-socket\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428352    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-cni-bin\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428374    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-cnibin\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428395    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-cni-bin\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-hostroot\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428441    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-conf-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428448    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-2x9wh\""
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428465    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428490    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-sys-fs\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.428536 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428528    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhk78\" (UniqueName: \"kubernetes.io/projected/4dcaef25-bb54-46c5-8973-8d4dd38deb93-kube-api-access-jhk78\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428557    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-daemon-config\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428620    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-etc-kubernetes\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428648    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428674    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2x25\" (UniqueName: \"kubernetes.io/projected/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-kube-api-access-q2x25\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428704    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-system-cni-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428727    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428622    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428729    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-cni-binary-copy\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428795    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gltjt\" (UniqueName: \"kubernetes.io/projected/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-kube-api-access-gltjt\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-kubelet\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-run-netns\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428910    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-k8s-cni-cncf-io\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428967    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4dcaef25-bb54-46c5-8973-8d4dd38deb93-host\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.428989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-node-log\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.429012    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-env-overrides\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.429036    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovn-node-metrics-cert\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429889 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.429060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-kubelet\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429889 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.429084    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-multus-certs\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.429889 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.429110    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-etc-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.429889 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.429354    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.430158 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.430138    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Jul 24 20:15:27.430355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.430338    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Jul 24 20:15:27.430790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.430762    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.430855 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.430836    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:27.430984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.430960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zw9f6\""
Jul 24 20:15:27.432058 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.432023    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Jul 24 20:15:27.432257 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.432239    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Jul 24 20:15:27.432446 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.432429    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-hvdnn\""
Jul 24 20:15:27.432828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.432808    2566 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Jul 24 20:15:27.458574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.458510    2566 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-kl7mz"
Jul 24 20:15:27.466454 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.466429    2566 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-kl7mz"
Jul 24 20:15:27.515725 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.515630    2566 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Jul 24 20:15:27.530371 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530338    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-sys-fs\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.530371 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530376    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhk78\" (UniqueName: \"kubernetes.io/projected/4dcaef25-bb54-46c5-8973-8d4dd38deb93-kube-api-access-jhk78\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530398    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-daemon-config\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530428    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530455    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530474    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-sys-fs\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530479    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-cni-binary-copy\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530532    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530578    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-kubelet\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-run-netns\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.530714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-node-log\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530744    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovn-node-metrics-cert\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-multus-certs\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-run-netns\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530802    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s6www\" (UniqueName: \"kubernetes.io/projected/bda593d7-440f-45d9-a969-22ef7982c75c-kube-api-access-s6www\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530830    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-etc-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovnkube-script-lib\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-socket-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-device-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9659\" (UniqueName: \"kubernetes.io/projected/68c5be09-ca2b-4a6c-a273-85c70f8650dc-kube-api-access-j9659\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4dcaef25-bb54-46c5-8973-8d4dd38deb93-serviceca\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.530979    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-cni-netd\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531006    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531053    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovnkube-config\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531080    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-os-release\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531105    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysctl-conf\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531131    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-var-lib-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.531155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531160    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-netns\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531185    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-run\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531196    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-daemon-config\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531209    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-sys\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531256    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-tuned\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531274    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-cni-binary-copy\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531285    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5758874-fada-4336-8293-06d1b16dffa8-iptables-alerter-script\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531278    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531312    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-cnibin\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysctl-d\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531406    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-kubelet\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531524    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-etc-selinux\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531567    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-node-log\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-registration-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531660    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-log-socket\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531667    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-cni-netd\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531687    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-cni-bin\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531654    2566 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Jul 24 20:15:27.532074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531714    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-netns\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531762    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-cni-bin\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531764    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-etc-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531819    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-registration-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531862    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-log-socket\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/4dcaef25-bb54-46c5-8973-8d4dd38deb93-serviceca\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531865    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-os-release\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysconfig\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bda593d7-440f-45d9-a969-22ef7982c75c-hosts-file\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531933    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-device-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-var-lib-openvswitch\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.531983    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-cni-bin\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532023    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-cni-bin\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532113    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-socket-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-etc-kubernetes\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532193    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-lib-modules\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.532984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532253    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b98e30d-2603-4bda-8cc3-00c359c50035-tmp\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74hpx\" (UniqueName: \"kubernetes.io/projected/0b98e30d-2603-4bda-8cc3-00c359c50035-kube-api-access-74hpx\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532313    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-multus-certs\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532375    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovnkube-config\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532378    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda593d7-440f-45d9-a969-22ef7982c75c-tmp-dir\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532399    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-etc-kubernetes\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532427    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bf2nl\" (UniqueName: \"kubernetes.io/projected/f5758874-fada-4336-8293-06d1b16dffa8-kube-api-access-bf2nl\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-system-cni-dir\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532483    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2x25\" (UniqueName: \"kubernetes.io/projected/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-kube-api-access-q2x25\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532564    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-system-cni-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532579    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovnkube-script-lib\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532642    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532665    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-system-cni-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532708    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-k8s-cni-cncf-io\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532733    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-run-ovn-kubernetes\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5758874-fada-4336-8293-06d1b16dffa8-host-slash\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.533887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532814    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-run-k8s-cni-cncf-io\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532813    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v78kl\" (UniqueName: \"kubernetes.io/projected/8efc8d87-bd58-4611-b380-47f8ddf39e81-kube-api-access-v78kl\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532864    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4dcaef25-bb54-46c5-8973-8d4dd38deb93-host\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-env-overrides\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532916    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-kubelet\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532936    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/4dcaef25-bb54-46c5-8973-8d4dd38deb93-host\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532946    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.532992    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-kubelet\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533058    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-cni-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-socket-dir-parent\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533126    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/01d20d74-1284-4562-aaac-c289b038f9b9-kubelet-config\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d07a0502-809b-4214-886b-a43cced3a8c6-agent-certs\") pod \"konnectivity-agent-wd5fk\" (UID: \"d07a0502-809b-4214-886b-a43cced3a8c6\") " pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533178    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d07a0502-809b-4214-886b-a43cced3a8c6-konnectivity-ca\") pod \"konnectivity-agent-wd5fk\" (UID: \"d07a0502-809b-4214-886b-a43cced3a8c6\") " pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533204    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9qqb\" (UniqueName: \"kubernetes.io/projected/95607563-c0dd-46eb-a0cf-c1d1ec957617-kube-api-access-m9qqb\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533231    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-os-release\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-slash\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.534874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-systemd\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533304    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-env-overrides\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533362    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-cni-multus\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533385    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-cni-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533387    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-systemd\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-socket-dir-parent\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533521    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-host-var-lib-cni-multus\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533569    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/01d20d74-1284-4562-aaac-c289b038f9b9-dbus\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-systemd\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gltjt\" (UniqueName: \"kubernetes.io/projected/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-kube-api-access-gltjt\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533628    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-host-slash\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-systemd-units\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533798    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-ovn\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533831    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-cnibin\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533853    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-hostroot\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533855    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-systemd-units\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533873    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-conf-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.535801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533907    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-multus-conf-dir\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-run-ovn\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-hostroot\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533953    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-cnibin\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533957    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-modprobe-d\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.533990    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-kubernetes\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.534015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-var-lib-kubelet\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.534048    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-host\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.534072    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.534163    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/68c5be09-ca2b-4a6c-a273-85c70f8650dc-kubelet-dir\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.536786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.535432    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-ovn-node-metrics-cert\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.540142 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.539794    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhk78\" (UniqueName: \"kubernetes.io/projected/4dcaef25-bb54-46c5-8973-8d4dd38deb93-kube-api-access-jhk78\") pod \"node-ca-qt7hz\" (UID: \"4dcaef25-bb54-46c5-8973-8d4dd38deb93\") " pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.540142 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.539838    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 20:15:27.540142 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.539863    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 20:15:27.540142 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.539878    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gcfpb for pod openshift-network-diagnostics/network-check-target-bls72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:27.540142 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.539983    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb podName:d1eb465f-2b0c-400e-86c9-bf527b76e75a nodeName:}" failed. No retries permitted until 2026-07-24 20:15:28.039960884 +0000 UTC m=+2.966099772 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-gcfpb" (UniqueName: "kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb") pod "network-check-target-bls72" (UID: "d1eb465f-2b0c-400e-86c9-bf527b76e75a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:27.540936 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.540912    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9659\" (UniqueName: \"kubernetes.io/projected/68c5be09-ca2b-4a6c-a273-85c70f8650dc-kube-api-access-j9659\") pod \"aws-ebs-csi-driver-node-bfgvg\" (UID: \"68c5be09-ca2b-4a6c-a273-85c70f8650dc\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.541670 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.541645    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2x25\" (UniqueName: \"kubernetes.io/projected/6d9a3c32-50f1-47a3-823c-c8f345b47d7b-kube-api-access-q2x25\") pod \"ovnkube-node-5txqj\" (UID: \"6d9a3c32-50f1-47a3-823c-c8f345b47d7b\") " pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.542944 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.542869    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gltjt\" (UniqueName: \"kubernetes.io/projected/55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24-kube-api-access-gltjt\") pod \"multus-9kv2f\" (UID: \"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24\") " pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.635263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635231    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-modprobe-d\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-kubernetes\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635278    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-var-lib-kubelet\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-host\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-kubernetes\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635420    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-modprobe-d\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635437    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-var-lib-kubelet\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-host\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635507    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635558    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.635628    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635636    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s6www\" (UniqueName: \"kubernetes.io/projected/bda593d7-440f-45d9-a969-22ef7982c75c-kube-api-access-s6www\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635673    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysctl-conf\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.635698    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs podName:95607563-c0dd-46eb-a0cf-c1d1ec957617 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:28.135677901 +0000 UTC m=+3.061816789 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs") pod "network-metrics-daemon-qh4xx" (UID: "95607563-c0dd-46eb-a0cf-c1d1ec957617") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysctl-conf\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635852    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-run\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.635892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635887    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-sys\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635911    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-tuned\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635939    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5758874-fada-4336-8293-06d1b16dffa8-iptables-alerter-script\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-sys\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635966    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-cnibin\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.635975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-run\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636002    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-cnibin\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysctl-d\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysconfig\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bda593d7-440f-45d9-a969-22ef7982c75c-hosts-file\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636093    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-lib-modules\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636116    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b98e30d-2603-4bda-8cc3-00c359c50035-tmp\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636122    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysctl-d\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636139    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74hpx\" (UniqueName: \"kubernetes.io/projected/0b98e30d-2603-4bda-8cc3-00c359c50035-kube-api-access-74hpx\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636165    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda593d7-440f-45d9-a969-22ef7982c75c-tmp-dir\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636188    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bf2nl\" (UniqueName: \"kubernetes.io/projected/f5758874-fada-4336-8293-06d1b16dffa8-kube-api-access-bf2nl\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636216    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-system-cni-dir\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.636341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636219    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bda593d7-440f-45d9-a969-22ef7982c75c-hosts-file\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636244    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-sysconfig\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5758874-fada-4336-8293-06d1b16dffa8-host-slash\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636282    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v78kl\" (UniqueName: \"kubernetes.io/projected/8efc8d87-bd58-4611-b380-47f8ddf39e81-kube-api-access-v78kl\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636364    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/01d20d74-1284-4562-aaac-c289b038f9b9-kubelet-config\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636388    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d07a0502-809b-4214-886b-a43cced3a8c6-agent-certs\") pod \"konnectivity-agent-wd5fk\" (UID: \"d07a0502-809b-4214-886b-a43cced3a8c6\") " pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d07a0502-809b-4214-886b-a43cced3a8c6-konnectivity-ca\") pod \"konnectivity-agent-wd5fk\" (UID: \"d07a0502-809b-4214-886b-a43cced3a8c6\") " pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636464    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f5758874-fada-4336-8293-06d1b16dffa8-host-slash\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636533    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-lib-modules\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636576    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bda593d7-440f-45d9-a969-22ef7982c75c-tmp-dir\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636605    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/f5758874-fada-4336-8293-06d1b16dffa8-iptables-alerter-script\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636657    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9qqb\" (UniqueName: \"kubernetes.io/projected/95607563-c0dd-46eb-a0cf-c1d1ec957617-kube-api-access-m9qqb\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/01d20d74-1284-4562-aaac-c289b038f9b9-kubelet-config\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.637209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636698    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-os-release\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636742    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-systemd\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.636759    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/01d20d74-1284-4562-aaac-c289b038f9b9-dbus\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636773    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8efc8d87-bd58-4611-b380-47f8ddf39e81-cni-binary-copy\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:27.636813    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret podName:01d20d74-1284-4562-aaac-c289b038f9b9 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:28.136794286 +0000 UTC m=+3.062933174 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret") pod "global-pull-secret-syncer-gqxld" (UID: "01d20d74-1284-4562-aaac-c289b038f9b9") : object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636838    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636846    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-system-cni-dir\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/01d20d74-1284-4562-aaac-c289b038f9b9-dbus\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.636939    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-systemd\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.637033    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"os-release\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-os-release\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.637067    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8efc8d87-bd58-4611-b380-47f8ddf39e81-tuning-conf-dir\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.637968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.637718    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/d07a0502-809b-4214-886b-a43cced3a8c6-konnectivity-ca\") pod \"konnectivity-agent-wd5fk\" (UID: \"d07a0502-809b-4214-886b-a43cced3a8c6\") " pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.638717 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.638687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/0b98e30d-2603-4bda-8cc3-00c359c50035-etc-tuned\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.639155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.639133    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0b98e30d-2603-4bda-8cc3-00c359c50035-tmp\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.639684 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.639661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/d07a0502-809b-4214-886b-a43cced3a8c6-agent-certs\") pod \"konnectivity-agent-wd5fk\" (UID: \"d07a0502-809b-4214-886b-a43cced3a8c6\") " pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.646910 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.646114    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s6www\" (UniqueName: \"kubernetes.io/projected/bda593d7-440f-45d9-a969-22ef7982c75c-kube-api-access-s6www\") pod \"node-resolver-6gllf\" (UID: \"bda593d7-440f-45d9-a969-22ef7982c75c\") " pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:27.649688 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.649627    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74hpx\" (UniqueName: \"kubernetes.io/projected/0b98e30d-2603-4bda-8cc3-00c359c50035-kube-api-access-74hpx\") pod \"tuned-lvk7m\" (UID: \"0b98e30d-2603-4bda-8cc3-00c359c50035\") " pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.649688 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.649668    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bf2nl\" (UniqueName: \"kubernetes.io/projected/f5758874-fada-4336-8293-06d1b16dffa8-kube-api-access-bf2nl\") pod \"iptables-alerter-zpxbt\" (UID: \"f5758874-fada-4336-8293-06d1b16dffa8\") " pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.652292 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.652268    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v78kl\" (UniqueName: \"kubernetes.io/projected/8efc8d87-bd58-4611-b380-47f8ddf39e81-kube-api-access-v78kl\") pod \"multus-additional-cni-plugins-2pz67\" (UID: \"8efc8d87-bd58-4611-b380-47f8ddf39e81\") " pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.652509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.652482    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9qqb\" (UniqueName: \"kubernetes.io/projected/95607563-c0dd-46eb-a0cf-c1d1ec957617-kube-api-access-m9qqb\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:27.726311 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.726266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:27.734896 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.734870    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9kv2f"
Jul 24 20:15:27.745828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.745797    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg"
Jul 24 20:15:27.751923 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.751898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-qt7hz"
Jul 24 20:15:27.761109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.761085    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-zpxbt"
Jul 24 20:15:27.769975 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.769935    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:27.777932 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.777912    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-2pz67"
Jul 24 20:15:27.791157 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.791125    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-lvk7m"
Jul 24 20:15:27.797337 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:27.797303    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-6gllf"
Jul 24 20:15:28.040229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.040134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:28.040386 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.040286    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 20:15:28.040386 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.040307    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 20:15:28.040386 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.040316    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gcfpb for pod openshift-network-diagnostics/network-check-target-bls72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:28.040386 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.040372    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb podName:d1eb465f-2b0c-400e-86c9-bf527b76e75a nodeName:}" failed. No retries permitted until 2026-07-24 20:15:29.040358275 +0000 UTC m=+3.966497159 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcfpb" (UniqueName: "kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb") pod "network-check-target-bls72" (UID: "d1eb465f-2b0c-400e-86c9-bf527b76e75a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:28.141349 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.141310    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:28.141566 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.141385    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:28.141566 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.141502    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:28.141678 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.141602    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret podName:01d20d74-1284-4562-aaac-c289b038f9b9 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:29.141579891 +0000 UTC m=+4.067718782 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret") pod "global-pull-secret-syncer-gqxld" (UID: "01d20d74-1284-4562-aaac-c289b038f9b9") : object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:28.141678 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.141503    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:28.141678 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.141657    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs podName:95607563-c0dd-46eb-a0cf-c1d1ec957617 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:29.141642299 +0000 UTC m=+4.067781194 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs") pod "network-metrics-daemon-qh4xx" (UID: "95607563-c0dd-46eb-a0cf-c1d1ec957617") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:28.468880 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.468760    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 20:10:28 +0000 UTC" deadline="2028-02-21 12:32:34.913523895 +0000 UTC"
Jul 24 20:15:28.468880 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.468827    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13840h17m6.444702398s"
Jul 24 20:15:28.561303 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.561275    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod68c5be09_ca2b_4a6c_a273_85c70f8650dc.slice/crio-d4080790da798e3072657f27d8df097235afd0094041da521eb81102c77cd74f WatchSource:0}: Error finding container d4080790da798e3072657f27d8df097235afd0094041da521eb81102c77cd74f: Status 404 returned error can't find the container with id d4080790da798e3072657f27d8df097235afd0094041da521eb81102c77cd74f
Jul 24 20:15:28.563958 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.563819    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b98e30d_2603_4bda_8cc3_00c359c50035.slice/crio-f6726195347466cd387edfbdfd50348d993a03dd02dc6f3309e2da23b76d9bcc WatchSource:0}: Error finding container f6726195347466cd387edfbdfd50348d993a03dd02dc6f3309e2da23b76d9bcc: Status 404 returned error can't find the container with id f6726195347466cd387edfbdfd50348d993a03dd02dc6f3309e2da23b76d9bcc
Jul 24 20:15:28.564924 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.564898    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod55bcdbdd_73f7_4ebc_8a7b_7b42dcb4bb24.slice/crio-792242c022fec25d9f9864ca1f6bed4f21304e5316dc068417055814e2636e98 WatchSource:0}: Error finding container 792242c022fec25d9f9864ca1f6bed4f21304e5316dc068417055814e2636e98: Status 404 returned error can't find the container with id 792242c022fec25d9f9864ca1f6bed4f21304e5316dc068417055814e2636e98
Jul 24 20:15:28.569486 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.569462    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbda593d7_440f_45d9_a969_22ef7982c75c.slice/crio-8419d3384fd87a63f616661ec656ffb7fb621f79bd44bf2f439e2819221b06d8 WatchSource:0}: Error finding container 8419d3384fd87a63f616661ec656ffb7fb621f79bd44bf2f439e2819221b06d8: Status 404 returned error can't find the container with id 8419d3384fd87a63f616661ec656ffb7fb621f79bd44bf2f439e2819221b06d8
Jul 24 20:15:28.570348 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.570325    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d9a3c32_50f1_47a3_823c_c8f345b47d7b.slice/crio-e87c3c9c9ea3efd5446a94b2b0fc18f11a407158fac1ebbdc143edb3b6769a73 WatchSource:0}: Error finding container e87c3c9c9ea3efd5446a94b2b0fc18f11a407158fac1ebbdc143edb3b6769a73: Status 404 returned error can't find the container with id e87c3c9c9ea3efd5446a94b2b0fc18f11a407158fac1ebbdc143edb3b6769a73
Jul 24 20:15:28.571983 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.571615    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf5758874_fada_4336_8293_06d1b16dffa8.slice/crio-66b84fd7634fb2d0965f197672f7368a782fba30efa209f8b4ca6a6ee5036f74 WatchSource:0}: Error finding container 66b84fd7634fb2d0965f197672f7368a782fba30efa209f8b4ca6a6ee5036f74: Status 404 returned error can't find the container with id 66b84fd7634fb2d0965f197672f7368a782fba30efa209f8b4ca6a6ee5036f74
Jul 24 20:15:28.573225 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.573200    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dcaef25_bb54_46c5_8973_8d4dd38deb93.slice/crio-0401386f345c773c41e33109d5f9d093cec3b987a9915609fe33ca4bf0e722e5 WatchSource:0}: Error finding container 0401386f345c773c41e33109d5f9d093cec3b987a9915609fe33ca4bf0e722e5: Status 404 returned error can't find the container with id 0401386f345c773c41e33109d5f9d093cec3b987a9915609fe33ca4bf0e722e5
Jul 24 20:15:28.576260 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.574375    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8efc8d87_bd58_4611_b380_47f8ddf39e81.slice/crio-8b7801fcfef705113438c0c926b9a030ed0dee0ce9a398feedd5dd0adad8d216 WatchSource:0}: Error finding container 8b7801fcfef705113438c0c926b9a030ed0dee0ce9a398feedd5dd0adad8d216: Status 404 returned error can't find the container with id 8b7801fcfef705113438c0c926b9a030ed0dee0ce9a398feedd5dd0adad8d216
Jul 24 20:15:28.577221 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:15:28.577192    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07a0502_809b_4214_886b_a43cced3a8c6.slice/crio-0a9ca813641d1a36e1c025ccf8f0e19439c53c1653385a80ed36f4f70b52a956 WatchSource:0}: Error finding container 0a9ca813641d1a36e1c025ccf8f0e19439c53c1653385a80ed36f4f70b52a956: Status 404 returned error can't find the container with id 0a9ca813641d1a36e1c025ccf8f0e19439c53c1653385a80ed36f4f70b52a956
Jul 24 20:15:28.687151 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.687122    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:28.687292 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:28.687241    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:28.696598 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.696563    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal" event={"ID":"58be0bbdb2076ba8a2b4b2e2c5986370","Type":"ContainerStarted","Data":"4dc614a55c7a4ff6d15b2597e6cd27e1c735209a277e5df060ba3920faea5720"}
Jul 24 20:15:28.697924 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.697899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerStarted","Data":"8b7801fcfef705113438c0c926b9a030ed0dee0ce9a398feedd5dd0adad8d216"}
Jul 24 20:15:28.699176 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.699120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qt7hz" event={"ID":"4dcaef25-bb54-46c5-8973-8d4dd38deb93","Type":"ContainerStarted","Data":"0401386f345c773c41e33109d5f9d093cec3b987a9915609fe33ca4bf0e722e5"}
Jul 24 20:15:28.700598 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.700576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9kv2f" event={"ID":"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24","Type":"ContainerStarted","Data":"792242c022fec25d9f9864ca1f6bed4f21304e5316dc068417055814e2636e98"}
Jul 24 20:15:28.701657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.701637    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lvk7m" event={"ID":"0b98e30d-2603-4bda-8cc3-00c359c50035","Type":"ContainerStarted","Data":"f6726195347466cd387edfbdfd50348d993a03dd02dc6f3309e2da23b76d9bcc"}
Jul 24 20:15:28.703741 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.703711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg" event={"ID":"68c5be09-ca2b-4a6c-a273-85c70f8650dc","Type":"ContainerStarted","Data":"d4080790da798e3072657f27d8df097235afd0094041da521eb81102c77cd74f"}
Jul 24 20:15:28.705933 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.705901    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wd5fk" event={"ID":"d07a0502-809b-4214-886b-a43cced3a8c6","Type":"ContainerStarted","Data":"0a9ca813641d1a36e1c025ccf8f0e19439c53c1653385a80ed36f4f70b52a956"}
Jul 24 20:15:28.707676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.707657    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpxbt" event={"ID":"f5758874-fada-4336-8293-06d1b16dffa8","Type":"ContainerStarted","Data":"66b84fd7634fb2d0965f197672f7368a782fba30efa209f8b4ca6a6ee5036f74"}
Jul 24 20:15:28.709093 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.709072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"e87c3c9c9ea3efd5446a94b2b0fc18f11a407158fac1ebbdc143edb3b6769a73"}
Jul 24 20:15:28.710263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.710235    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gllf" event={"ID":"bda593d7-440f-45d9-a969-22ef7982c75c","Type":"ContainerStarted","Data":"8419d3384fd87a63f616661ec656ffb7fb621f79bd44bf2f439e2819221b06d8"}
Jul 24 20:15:28.715175 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:28.715068    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-245.ec2.internal" podStartSLOduration=1.715055099 podStartE2EDuration="1.715055099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:28.715007108 +0000 UTC m=+3.641146035" watchObservedRunningTime="2026-07-24 20:15:28.715055099 +0000 UTC m=+3.641194008"
Jul 24 20:15:29.055289 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.055126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:29.062096 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.057740    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 20:15:29.062096 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.057780    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 20:15:29.062096 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.057805    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gcfpb for pod openshift-network-diagnostics/network-check-target-bls72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:29.062096 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.057900    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb podName:d1eb465f-2b0c-400e-86c9-bf527b76e75a nodeName:}" failed. No retries permitted until 2026-07-24 20:15:31.057870064 +0000 UTC m=+5.984008951 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcfpb" (UniqueName: "kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb") pod "network-check-target-bls72" (UID: "d1eb465f-2b0c-400e-86c9-bf527b76e75a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:29.156513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.156472    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:29.156710 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.156584    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:29.156776 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.156758    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:29.156828 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.156823    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs podName:95607563-c0dd-46eb-a0cf-c1d1ec957617 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:31.156803821 +0000 UTC m=+6.082942708 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs") pod "network-metrics-daemon-qh4xx" (UID: "95607563-c0dd-46eb-a0cf-c1d1ec957617") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:29.157268 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.157243    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:29.157342 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.157306    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret podName:01d20d74-1284-4562-aaac-c289b038f9b9 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:31.157290029 +0000 UTC m=+6.083428915 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret") pod "global-pull-secret-syncer-gqxld" (UID: "01d20d74-1284-4562-aaac-c289b038f9b9") : object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:29.469840 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.469714    2566 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-07-23 20:10:28 +0000 UTC" deadline="2028-01-23 23:33:13.220681624 +0000 UTC"
Jul 24 20:15:29.469840 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.469787    2566 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13155h17m43.750900059s"
Jul 24 20:15:29.513025 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.512991    2566 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Jul 24 20:15:29.689816 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.688975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:29.689816 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.689104    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:29.689816 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.689565    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:29.689816 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:29.689694    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:29.743105 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.741781    2566 generic.go:356] "Generic (PLEG): container finished" podID="2e0b7853205bd7e1e30de35767acd8e1" containerID="8eb5d362a56d3e19b90084075ae658ec52a692960b44c06433142098c7fee43f" exitCode=0
Jul 24 20:15:29.743105 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:29.742766    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" event={"ID":"2e0b7853205bd7e1e30de35767acd8e1","Type":"ContainerDied","Data":"8eb5d362a56d3e19b90084075ae658ec52a692960b44c06433142098c7fee43f"}
Jul 24 20:15:30.686627 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:30.686583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:30.687118 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:30.686733    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:30.766657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:30.765928    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" event={"ID":"2e0b7853205bd7e1e30de35767acd8e1","Type":"ContainerStarted","Data":"3432113206cafbb659c4450e39e491c8d53c7daae09b68ef4e0b7bbf2392c650"}
Jul 24 20:15:31.076019 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:31.075927    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:31.076197 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.076110    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 20:15:31.076197 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.076131    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 20:15:31.076197 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.076148    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gcfpb for pod openshift-network-diagnostics/network-check-target-bls72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:31.076380 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.076211    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb podName:d1eb465f-2b0c-400e-86c9-bf527b76e75a nodeName:}" failed. No retries permitted until 2026-07-24 20:15:35.076191266 +0000 UTC m=+10.002330152 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcfpb" (UniqueName: "kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb") pod "network-check-target-bls72" (UID: "d1eb465f-2b0c-400e-86c9-bf527b76e75a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:31.177377 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:31.177321    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:31.177572 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:31.177407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:31.177572 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.177535    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:31.177716 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.177619    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret podName:01d20d74-1284-4562-aaac-c289b038f9b9 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:35.177599018 +0000 UTC m=+10.103737907 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret") pod "global-pull-secret-syncer-gqxld" (UID: "01d20d74-1284-4562-aaac-c289b038f9b9") : object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:31.177716 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.177694    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:31.177835 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.177734    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs podName:95607563-c0dd-46eb-a0cf-c1d1ec957617 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:35.177720034 +0000 UTC m=+10.103858921 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs") pod "network-metrics-daemon-qh4xx" (UID: "95607563-c0dd-46eb-a0cf-c1d1ec957617") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:31.687194 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:31.686669    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:31.687194 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.686810    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:31.690215 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:31.687856    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:31.690215 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:31.687973    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:32.686309 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:32.686268    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:32.686499 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:32.686427    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:33.687481 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:33.687337    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:33.687481 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:33.687371    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:33.688057 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:33.687488    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:33.688057 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:33.687822    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:34.686923 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:34.686882    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:34.687115 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:34.687040    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:35.110586 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:35.110457    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:35.111075 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.110697    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 20:15:35.111075 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.110720    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 20:15:35.111075 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.110735    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gcfpb for pod openshift-network-diagnostics/network-check-target-bls72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:35.111075 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.110797    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb podName:d1eb465f-2b0c-400e-86c9-bf527b76e75a nodeName:}" failed. No retries permitted until 2026-07-24 20:15:43.11077856 +0000 UTC m=+18.036917447 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcfpb" (UniqueName: "kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb") pod "network-check-target-bls72" (UID: "d1eb465f-2b0c-400e-86c9-bf527b76e75a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:35.212579 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:35.211870    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:35.212579 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:35.211954    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:35.212579 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.212131    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:35.212579 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.212203    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs podName:95607563-c0dd-46eb-a0cf-c1d1ec957617 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:43.212180987 +0000 UTC m=+18.138319875 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs") pod "network-metrics-daemon-qh4xx" (UID: "95607563-c0dd-46eb-a0cf-c1d1ec957617") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:35.212987 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.212622    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:35.212987 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.212676    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret podName:01d20d74-1284-4562-aaac-c289b038f9b9 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:43.212661179 +0000 UTC m=+18.138800065 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret") pod "global-pull-secret-syncer-gqxld" (UID: "01d20d74-1284-4562-aaac-c289b038f9b9") : object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:35.688103 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:35.688062    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:35.688295 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.688181    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:35.688295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:35.688266    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:35.688416 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:35.688359    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:36.687178 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:36.686627    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:36.687178 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:36.686768    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:37.686284 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:37.686250    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:37.686514 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:37.686260    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:37.686514 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:37.686389    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:37.686514 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:37.686422    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:38.686386 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:38.686351    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:38.686857 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:38.686511    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:39.686447 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:39.686408    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:39.686447 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:39.686447    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:39.687181 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:39.687145    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:39.687294 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:39.687261    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:40.686652 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:40.686616    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:40.687067 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:40.686765    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:41.686634 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:41.686599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:41.686801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:41.686599    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:41.686801 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:41.686744    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:41.687218 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:41.686808    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:42.686369 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:42.686327    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:42.686615 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:42.686459    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:43.177962 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:43.177885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:43.178431 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.178018    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 20:15:43.178431 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.178037    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 20:15:43.178431 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.178051    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gcfpb for pod openshift-network-diagnostics/network-check-target-bls72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:43.178431 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.178111    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb podName:d1eb465f-2b0c-400e-86c9-bf527b76e75a nodeName:}" failed. No retries permitted until 2026-07-24 20:15:59.178094169 +0000 UTC m=+34.104233074 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcfpb" (UniqueName: "kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb") pod "network-check-target-bls72" (UID: "d1eb465f-2b0c-400e-86c9-bf527b76e75a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:43.278814 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:43.278771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:43.278996 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:43.278851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:43.278996 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.278917    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:43.278996 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.278938    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:43.278996 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.278982    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret podName:01d20d74-1284-4562-aaac-c289b038f9b9 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:59.27896365 +0000 UTC m=+34.205102538 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret") pod "global-pull-secret-syncer-gqxld" (UID: "01d20d74-1284-4562-aaac-c289b038f9b9") : object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:43.278996 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.278999    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs podName:95607563-c0dd-46eb-a0cf-c1d1ec957617 nodeName:}" failed. No retries permitted until 2026-07-24 20:15:59.278991009 +0000 UTC m=+34.205129904 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs") pod "network-metrics-daemon-qh4xx" (UID: "95607563-c0dd-46eb-a0cf-c1d1ec957617") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:43.686601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:43.686570    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:43.686601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:43.686595    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:43.686833 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.686707    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:43.686833 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:43.686819    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:44.687172 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:44.687131    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:44.687590 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:44.687283    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:45.689284 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:45.689252    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:45.689687 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:45.689365    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:45.689687 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:45.689397    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:45.689687 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:45.689453    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:46.687114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:46.687084    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:46.687274 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:46.687185    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:47.687242 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.687008    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:47.688020 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.687015    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:47.688020 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:47.687351    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:47.688020 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:47.687424    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:47.802471 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.802390    2566 generic.go:356] "Generic (PLEG): container finished" podID="8efc8d87-bd58-4611-b380-47f8ddf39e81" containerID="0ea9e48e4f323208ccab945fd482f4231d0aad6a179f10f04e5b58a676bf13fb" exitCode=0
Jul 24 20:15:47.802647 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.802492    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerDied","Data":"0ea9e48e4f323208ccab945fd482f4231d0aad6a179f10f04e5b58a676bf13fb"}
Jul 24 20:15:47.804894 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.804751    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-qt7hz" event={"ID":"4dcaef25-bb54-46c5-8973-8d4dd38deb93","Type":"ContainerStarted","Data":"9b3016f630dde6e7daa0fad87820603d962169752ec0f1fab57f0ed994a026e8"}
Jul 24 20:15:47.806589 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.806556    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9kv2f" event={"ID":"55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24","Type":"ContainerStarted","Data":"2d545449fa29ba2faa8045ec652f983ac12ea006bacb209cbdf40600ab09e7d7"}
Jul 24 20:15:47.808539 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.808479    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-lvk7m" event={"ID":"0b98e30d-2603-4bda-8cc3-00c359c50035","Type":"ContainerStarted","Data":"dec4ef101e38bd35f29b5d481c61e7834789c384d3a522725c66845ea7540dfa"}
Jul 24 20:15:47.812902 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.811846    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg" event={"ID":"68c5be09-ca2b-4a6c-a273-85c70f8650dc","Type":"ContainerStarted","Data":"e7a863b535cd013c3edb7422d8e96ff9986dc0d29e6f6d35dee45c65a3806fbf"}
Jul 24 20:15:47.814351 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.814324    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-wd5fk" event={"ID":"d07a0502-809b-4214-886b-a43cced3a8c6","Type":"ContainerStarted","Data":"7ccf7949605e1c0b309cdccad2e02482adaa8a878bce0ece363045f3b1aca1e0"}
Jul 24 20:15:47.817711 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.817678    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"ae68439d30f3a774275d9ca87f383c7ebd3e1961d6e56f6343135a2b0a75c3ea"}
Jul 24 20:15:47.817827 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.817749    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"85bea24412f0e5ce64eff9e4daa92c9f1765f5936bd66c48b6f5ec4171d75a83"}
Jul 24 20:15:47.817827 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.817795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"0a96ea679a75591c4c33d91e8129305e58517f80154abe052f1f6c4c824116c2"}
Jul 24 20:15:47.817939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.817841    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"a68f16c6ae3388cd29a48489fb68ca54ec51426aba3aafc6cd7aec8164468698"}
Jul 24 20:15:47.817939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.817887    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"61e0af42c0d91b580079c1c4bd90114de647d347491b6e25b999ea6ca6438970"}
Jul 24 20:15:47.817939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.817934    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"26ebb12c8312d9e584ddb201e32938f11dcae731a64b4cb46bc8fbb7972bdc45"}
Jul 24 20:15:47.819054 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.819031    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-6gllf" event={"ID":"bda593d7-440f-45d9-a969-22ef7982c75c","Type":"ContainerStarted","Data":"b5e76592741f041e4e94225504a961e71df41152e9766d9c6805b284164dac09"}
Jul 24 20:15:47.826200 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.826162    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-245.ec2.internal" podStartSLOduration=20.826150335 podStartE2EDuration="20.826150335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:30.78507267 +0000 UTC m=+5.711211579" watchObservedRunningTime="2026-07-24 20:15:47.826150335 +0000 UTC m=+22.752289238"
Jul 24 20:15:47.840538 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.840488    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-wd5fk" podStartSLOduration=7.603849802 podStartE2EDuration="21.840475116s" totalImagesPullingTime="14.236625314s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:47.84018322 +0000 UTC m=+22.766322124" watchObservedRunningTime="2026-07-24 20:15:47.840475116 +0000 UTC m=+22.766614020"
Jul 24 20:15:47.856142 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.856105    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-lvk7m" podStartSLOduration=3.352218387 podStartE2EDuration="21.856091777s" totalImagesPullingTime="18.50387339s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:47.855654188 +0000 UTC m=+22.781793092" watchObservedRunningTime="2026-07-24 20:15:47.856091777 +0000 UTC m=+22.782230681"
Jul 24 20:15:47.870727 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.870690    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-6gllf" podStartSLOduration=3.373067727 podStartE2EDuration="21.870678179s" totalImagesPullingTime="18.497610452s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:47.870265772 +0000 UTC m=+22.796404676" watchObservedRunningTime="2026-07-24 20:15:47.870678179 +0000 UTC m=+22.796817082"
Jul 24 20:15:47.902158 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.902106    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-9kv2f" podStartSLOduration=3.151064641 podStartE2EDuration="21.902085377s" totalImagesPullingTime="18.751020736s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:47.88822027 +0000 UTC m=+22.814359173" watchObservedRunningTime="2026-07-24 20:15:47.902085377 +0000 UTC m=+22.828224283"
Jul 24 20:15:47.902557 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.902508    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-qt7hz" podStartSLOduration=3.41048256 podStartE2EDuration="21.902499983s" totalImagesPullingTime="18.492017423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:47.901694395 +0000 UTC m=+22.827833319" watchObservedRunningTime="2026-07-24 20:15:47.902499983 +0000 UTC m=+22.828638888"
Jul 24 20:15:47.929055 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.929021    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:47.929590 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:47.929571    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:48.047179 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.047148    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:48.108914 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.108877    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:48.432601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.432407    2566 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Jul 24 20:15:48.481248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.481139    2566 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-07-24T20:15:48.432596782Z","UUID":"9d836b23-e31e-46c5-ba83-698f52e7585b","Handler":null,"Name":"","Endpoint":""}
Jul 24 20:15:48.482931 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.482906    2566 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Jul 24 20:15:48.483038 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.482949    2566 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Jul 24 20:15:48.686895 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.686818    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:48.687066 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:48.686962    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:48.780117 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.780084    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:48.823716 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.823681    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg" event={"ID":"68c5be09-ca2b-4a6c-a273-85c70f8650dc","Type":"ContainerStarted","Data":"28f49b42e327ca037c5115a6d053b30646bb47aeabd5933a6cb2ab9472507487"}
Jul 24 20:15:48.825290 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.825263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-zpxbt" event={"ID":"f5758874-fada-4336-8293-06d1b16dffa8","Type":"ContainerStarted","Data":"39c667a73ea18bf93ca5afdb7001010673fa758b9d672320500d4652e74e40cc"}
Jul 24 20:15:48.826069 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.826049    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:48.826172 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.826110    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-wd5fk"
Jul 24 20:15:48.840293 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.840251    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-zpxbt" podStartSLOduration=4.346414284 podStartE2EDuration="22.840237916s" totalImagesPullingTime="18.493823632s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:48.839927409 +0000 UTC m=+23.766066311" watchObservedRunningTime="2026-07-24 20:15:48.840237916 +0000 UTC m=+23.766376820"
Jul 24 20:15:48.844577 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.844554    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:48.908863 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:48.908821    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:49.009123 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.009038    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:49.064360 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.064318    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:49.686681 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.686602    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:49.686681 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.686618    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:49.686945 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:49.686712    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:49.686945 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:49.686854    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:49.794792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.794760    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:49.829925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.829896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg" event={"ID":"68c5be09-ca2b-4a6c-a273-85c70f8650dc","Type":"ContainerStarted","Data":"12d905bcd6386e0b9c88af13e3dbfb38e06164a48d5e9a952408dfe66100d944"}
Jul 24 20:15:49.850590 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.850517    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-bfgvg" podStartSLOduration=3.328457253 podStartE2EDuration="23.850480836s" totalImagesPullingTime="20.522023583s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:49.848635295 +0000 UTC m=+24.774774223" watchObservedRunningTime="2026-07-24 20:15:49.850480836 +0000 UTC m=+24.776619739"
Jul 24 20:15:49.894438 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:49.894403    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:50.396278 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:50.396244    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:50.687210 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:50.687133    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:50.687363 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:50.687260    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:50.755501 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:50.755468    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:50.836613 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:50.836576    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"4de949914a0463764085b1edfcc23dc253853f6b937eb7f3ed9a054a90e2960e"}
Jul 24 20:15:50.855657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:50.855624    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:51.686454 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:51.686425    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:51.686653 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:51.686571    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:51.686653 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:51.686432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:51.686771 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:51.686742    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:52.687098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:52.686874    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:52.687694 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:52.687129    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:52.843640 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:52.843605    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" event={"ID":"6d9a3c32-50f1-47a3-823c-c8f345b47d7b","Type":"ContainerStarted","Data":"4981beeabde7357413da831fbf46aeb83b735db008e2006e8e7afad9f6d25690"}
Jul 24 20:15:52.843857 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:52.843831    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:52.845377 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:52.845356    2566 generic.go:356] "Generic (PLEG): container finished" podID="8efc8d87-bd58-4611-b380-47f8ddf39e81" containerID="d0e617932b47c60182ec88ced4432fb02c44fef6f3a10372df306217843cecfc" exitCode=0
Jul 24 20:15:52.845477 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:52.845395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerDied","Data":"d0e617932b47c60182ec88ced4432fb02c44fef6f3a10372df306217843cecfc"}
Jul 24 20:15:52.859389 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:52.859370    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:52.875280 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:52.875237    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj" podStartSLOduration=8.341158028 podStartE2EDuration="26.875225034s" totalImagesPullingTime="18.534067006s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:15:52.874778566 +0000 UTC m=+27.800917500" watchObservedRunningTime="2026-07-24 20:15:52.875225034 +0000 UTC m=+27.801363985"
Jul 24 20:15:53.687144 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:53.687070    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:53.687144 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:53.687102    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:53.687506 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:53.687210    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:53.687506 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:53.687318    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:53.847765 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:53.847739    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:53.847902 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:53.847810    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:53.862126 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:53.862099    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:15:54.192242 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.191950    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqxld"]
Jul 24 20:15:54.192462 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.192387    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:54.192557 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:54.192489    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:54.193261 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.193236    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qh4xx"]
Jul 24 20:15:54.193388 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.193348    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:54.193468 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:54.193448    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:54.204840 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.204779    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bls72"]
Jul 24 20:15:54.204932 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.204877    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:54.204982 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:54.204963    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:54.851545 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.851491    2566 generic.go:356] "Generic (PLEG): container finished" podID="8efc8d87-bd58-4611-b380-47f8ddf39e81" containerID="050f33acc4faea70ff022820a5b3b98055e8f7fc16d72b2d41a7af0f1c39a900" exitCode=0
Jul 24 20:15:54.852163 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:54.851558    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerDied","Data":"050f33acc4faea70ff022820a5b3b98055e8f7fc16d72b2d41a7af0f1c39a900"}
Jul 24 20:15:55.687720 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:55.687690    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:55.687877 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:55.687792    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:55.687925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:55.687882    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:55.688010 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:55.687996    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:55.688060 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:55.688025    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:55.688096 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:55.688069    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:56.127990 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:56.127956    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:56.228695 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:56.228615    2566 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log" limit=1024
Jul 24 20:15:56.858655 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:56.858618    2566 generic.go:356] "Generic (PLEG): container finished" podID="8efc8d87-bd58-4611-b380-47f8ddf39e81" containerID="df69633497767f2e6a0c76aaa4dce3967897689d2b4ff6186d840eab551c3a33" exitCode=0
Jul 24 20:15:56.859072 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:56.858665    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerDied","Data":"df69633497767f2e6a0c76aaa4dce3967897689d2b4ff6186d840eab551c3a33"}
Jul 24 20:15:57.686639 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:57.686607    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:57.687212 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:57.686606    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:57.687212 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:57.686730    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:57.687212 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:57.686621    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:57.687212 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:57.686780    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:57.687212 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:57.686879    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:59.197515 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.197473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:59.197924 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.197638    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Jul 24 20:15:59.197924 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.197654    2566 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Jul 24 20:15:59.197924 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.197664    2566 projected.go:196] Error preparing data for projected volume kube-api-access-gcfpb for pod openshift-network-diagnostics/network-check-target-bls72: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:59.197924 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.197715    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb podName:d1eb465f-2b0c-400e-86c9-bf527b76e75a nodeName:}" failed. No retries permitted until 2026-07-24 20:16:31.197699877 +0000 UTC m=+66.123838779 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-gcfpb" (UniqueName: "kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb") pod "network-check-target-bls72" (UID: "d1eb465f-2b0c-400e-86c9-bf527b76e75a") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Jul 24 20:15:59.298368 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.298330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:59.298550 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.298422    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:59.298550 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.298499    2566 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:59.298636 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.298570    2566 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:59.298636 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.298578    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs podName:95607563-c0dd-46eb-a0cf-c1d1ec957617 nodeName:}" failed. No retries permitted until 2026-07-24 20:16:31.298560545 +0000 UTC m=+66.224699439 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs") pod "network-metrics-daemon-qh4xx" (UID: "95607563-c0dd-46eb-a0cf-c1d1ec957617") : object "openshift-multus"/"metrics-daemon-secret" not registered
Jul 24 20:15:59.298636 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.298622    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret podName:01d20d74-1284-4562-aaac-c289b038f9b9 nodeName:}" failed. No retries permitted until 2026-07-24 20:16:31.298606289 +0000 UTC m=+66.224745175 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret") pod "global-pull-secret-syncer-gqxld" (UID: "01d20d74-1284-4562-aaac-c289b038f9b9") : object "kube-system"/"original-pull-secret" not registered
Jul 24 20:15:59.444774 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.444743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gllf_bda593d7-440f-45d9-a969-22ef7982c75c/dns-node-resolver/0.log"
Jul 24 20:15:59.687371 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.687332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:15:59.687584 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.687332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:15:59.687584 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.687485    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-qh4xx" podUID="95607563-c0dd-46eb-a0cf-c1d1ec957617"
Jul 24 20:15:59.687584 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.687332    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:15:59.687728 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.687589    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-bls72" podUID="d1eb465f-2b0c-400e-86c9-bf527b76e75a"
Jul 24 20:15:59.687728 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:15:59.687629    2566 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-gqxld" podUID="01d20d74-1284-4562-aaac-c289b038f9b9"
Jul 24 20:15:59.926423 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.926223    2566 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-245.ec2.internal" event="NodeReady"
Jul 24 20:15:59.926602 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.926559    2566 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Jul 24 20:15:59.980750 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.980633    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-76cbf5c457-pzkvq"]
Jul 24 20:15:59.983075 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.983054    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:15:59.992023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.991926    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Jul 24 20:15:59.992256 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.992234    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Jul 24 20:15:59.993178 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.993157    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-gpp76\""
Jul 24 20:15:59.994710 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.994692    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Jul 24 20:15:59.996648 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:15:59.996626    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Jul 24 20:16:00.001588 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.001571    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-7wtlx"]
Jul 24 20:16:00.003271 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.003254    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.006373 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.006357    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Jul 24 20:16:00.007283 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.007154    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-mgpkh\""
Jul 24 20:16:00.007283 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.007173    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Jul 24 20:16:00.008711 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.008690    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-2bds9"]
Jul 24 20:16:00.010545 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.010507    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76cbf5c457-pzkvq"]
Jul 24 20:16:00.010668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.010652    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.018563 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.018498    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-p92ks\""
Jul 24 20:16:00.018665 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.018500    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Jul 24 20:16:00.018800 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.018785    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Jul 24 20:16:00.022327 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.022308    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Jul 24 20:16:00.026349 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.026327    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7wtlx"]
Jul 24 20:16:00.029931 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.029910    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bds9"]
Jul 24 20:16:00.105777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.105746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58dc6394-715b-4b2a-a69f-29f9e21c81d9-ca-trust-extracted\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.105777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.105789    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hd5z8\" (UniqueName: \"kubernetes.io/projected/da0cd617-b89e-4636-bc16-eee42b32327b-kube-api-access-hd5z8\") pod \"ingress-canary-2bds9\" (UID: \"da0cd617-b89e-4636-bc16-eee42b32327b\") " pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.105993 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.105810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcf5q\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-kube-api-access-dcf5q\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.105993 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.105880    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-trusted-ca\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.105993 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.105943    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df2d6053-ad84-46c0-8cf6-60497f26536f-config-volume\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.105993 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.105970    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-image-registry-private-configuration\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.106181 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106003    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.106181 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106091    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-certificates\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.106181 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106162    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da0cd617-b89e-4636-bc16-eee42b32327b-cert\") pod \"ingress-canary-2bds9\" (UID: \"da0cd617-b89e-4636-bc16-eee42b32327b\") " pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.106299 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106189    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkj2d\" (UniqueName: \"kubernetes.io/projected/df2d6053-ad84-46c0-8cf6-60497f26536f-kube-api-access-nkj2d\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.106299 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106217    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-bound-sa-token\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.106299 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.106387 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106315    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6053-ad84-46c0-8cf6-60497f26536f-tmp-dir\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.106387 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.106360    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-installation-pull-secrets\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.206689 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206653    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58dc6394-715b-4b2a-a69f-29f9e21c81d9-ca-trust-extracted\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.206689 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206697    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hd5z8\" (UniqueName: \"kubernetes.io/projected/da0cd617-b89e-4636-bc16-eee42b32327b-kube-api-access-hd5z8\") pod \"ingress-canary-2bds9\" (UID: \"da0cd617-b89e-4636-bc16-eee42b32327b\") " pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcf5q\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-kube-api-access-dcf5q\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206750    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-trusted-ca\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df2d6053-ad84-46c0-8cf6-60497f26536f-config-volume\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206811    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-image-registry-private-configuration\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206839    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206934    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-certificates\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.206996    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da0cd617-b89e-4636-bc16-eee42b32327b-cert\") pod \"ingress-canary-2bds9\" (UID: \"da0cd617-b89e-4636-bc16-eee42b32327b\") " pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.207077    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.207098    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cbf5c457-pzkvq: secret "image-registry-tls" not found
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.207179    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls podName:58dc6394-715b-4b2a-a69f-29f9e21c81d9 nodeName:}" failed. No retries permitted until 2026-07-24 20:16:00.707159681 +0000 UTC m=+35.633298580 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls") pod "image-registry-76cbf5c457-pzkvq" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9") : secret "image-registry-tls" not found
Jul 24 20:16:00.207214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207168    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58dc6394-715b-4b2a-a69f-29f9e21c81d9-ca-trust-extracted\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207340    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkj2d\" (UniqueName: \"kubernetes.io/projected/df2d6053-ad84-46c0-8cf6-60497f26536f-kube-api-access-nkj2d\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207377    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-bound-sa-token\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207444    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6053-ad84-46c0-8cf6-60497f26536f-tmp-dir\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207451    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-certificates\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/df2d6053-ad84-46c0-8cf6-60497f26536f-config-volume\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-installation-pull-secrets\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.207643    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.207707    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls podName:df2d6053-ad84-46c0-8cf6-60497f26536f nodeName:}" failed. No retries permitted until 2026-07-24 20:16:00.707690112 +0000 UTC m=+35.633829006 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls") pod "dns-default-7wtlx" (UID: "df2d6053-ad84-46c0-8cf6-60497f26536f") : secret "dns-default-metrics-tls" not found
Jul 24 20:16:00.207776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df2d6053-ad84-46c0-8cf6-60497f26536f-tmp-dir\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.208187 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.207818    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-trusted-ca\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.212178 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.212153    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-image-registry-private-configuration\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.212310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.212184    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/da0cd617-b89e-4636-bc16-eee42b32327b-cert\") pod \"ingress-canary-2bds9\" (UID: \"da0cd617-b89e-4636-bc16-eee42b32327b\") " pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.212310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.212187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-installation-pull-secrets\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.238845 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.238783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkj2d\" (UniqueName: \"kubernetes.io/projected/df2d6053-ad84-46c0-8cf6-60497f26536f-kube-api-access-nkj2d\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.241612 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.241581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hd5z8\" (UniqueName: \"kubernetes.io/projected/da0cd617-b89e-4636-bc16-eee42b32327b-kube-api-access-hd5z8\") pod \"ingress-canary-2bds9\" (UID: \"da0cd617-b89e-4636-bc16-eee42b32327b\") " pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.242211 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.242166    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-bound-sa-token\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.242512 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.242485    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcf5q\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-kube-api-access-dcf5q\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.321696 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.321661    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-2bds9"
Jul 24 20:16:00.663925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.663885    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qt7hz_4dcaef25-bb54-46c5-8973-8d4dd38deb93/node-ca/0.log"
Jul 24 20:16:00.712372 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.712337    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:00.712566 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:00.712409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:00.712566 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.712495    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 20:16:00.712566 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.712517    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cbf5c457-pzkvq: secret "image-registry-tls" not found
Jul 24 20:16:00.712566 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.712550    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 20:16:00.712730 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.712593    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls podName:58dc6394-715b-4b2a-a69f-29f9e21c81d9 nodeName:}" failed. No retries permitted until 2026-07-24 20:16:01.712574005 +0000 UTC m=+36.638712891 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls") pod "image-registry-76cbf5c457-pzkvq" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9") : secret "image-registry-tls" not found
Jul 24 20:16:00.712730 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:00.712607    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls podName:df2d6053-ad84-46c0-8cf6-60497f26536f nodeName:}" failed. No retries permitted until 2026-07-24 20:16:01.712601478 +0000 UTC m=+36.638740366 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls") pod "dns-default-7wtlx" (UID: "df2d6053-ad84-46c0-8cf6-60497f26536f") : secret "dns-default-metrics-tls" not found
Jul 24 20:16:01.002184 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.002083    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-2bds9"]
Jul 24 20:16:01.005867 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:01.005805    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0cd617_b89e_4636_bc16_eee42b32327b.slice/crio-c6135f43c98c4037d7167f0ae0585989e5c5b44fb3b411e163aab65f0d884521 WatchSource:0}: Error finding container c6135f43c98c4037d7167f0ae0585989e5c5b44fb3b411e163aab65f0d884521: Status 404 returned error can't find the container with id c6135f43c98c4037d7167f0ae0585989e5c5b44fb3b411e163aab65f0d884521
Jul 24 20:16:01.686859 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.686827    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:16:01.687288 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.686826    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:16:01.687288 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.687012    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:16:01.691235 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.691190    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 20:16:01.692268 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.692250    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sm2r\""
Jul 24 20:16:01.692490 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.692470    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 20:16:01.692604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.692536    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 20:16:01.692668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.692599    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 20:16:01.692668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.692599    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b8xdt\""
Jul 24 20:16:01.719060 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.719015    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:01.719182 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.719106    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:01.719182 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:01.719121    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 20:16:01.719182 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:01.719182    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls podName:df2d6053-ad84-46c0-8cf6-60497f26536f nodeName:}" failed. No retries permitted until 2026-07-24 20:16:03.719165859 +0000 UTC m=+38.645304746 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls") pod "dns-default-7wtlx" (UID: "df2d6053-ad84-46c0-8cf6-60497f26536f") : secret "dns-default-metrics-tls" not found
Jul 24 20:16:01.719364 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:01.719224    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 20:16:01.719364 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:01.719238    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cbf5c457-pzkvq: secret "image-registry-tls" not found
Jul 24 20:16:01.719364 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:01.719290    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls podName:58dc6394-715b-4b2a-a69f-29f9e21c81d9 nodeName:}" failed. No retries permitted until 2026-07-24 20:16:03.719275563 +0000 UTC m=+38.645414450 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls") pod "image-registry-76cbf5c457-pzkvq" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9") : secret "image-registry-tls" not found
Jul 24 20:16:01.872734 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.872501    2566 generic.go:356] "Generic (PLEG): container finished" podID="8efc8d87-bd58-4611-b380-47f8ddf39e81" containerID="50b2b096199dd8d8e7abbced858c53807ebc757658f4f9143d59faf73d6cf4be" exitCode=0
Jul 24 20:16:01.872906 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.872579    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerDied","Data":"50b2b096199dd8d8e7abbced858c53807ebc757658f4f9143d59faf73d6cf4be"}
Jul 24 20:16:01.874046 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:01.874019    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bds9" event={"ID":"da0cd617-b89e-4636-bc16-eee42b32327b","Type":"ContainerStarted","Data":"c6135f43c98c4037d7167f0ae0585989e5c5b44fb3b411e163aab65f0d884521"}
Jul 24 20:16:02.880797 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:02.880762    2566 generic.go:356] "Generic (PLEG): container finished" podID="8efc8d87-bd58-4611-b380-47f8ddf39e81" containerID="4a970a0594511bc2ddf19378c5328b584c778eb15843f6dcf26b175b1bb6c572" exitCode=0
Jul 24 20:16:02.881272 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:02.880847    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerDied","Data":"4a970a0594511bc2ddf19378c5328b584c778eb15843f6dcf26b175b1bb6c572"}
Jul 24 20:16:03.734088 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:03.734052    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:03.734251 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:03.734114    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:03.734251 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:03.734203    2566 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Jul 24 20:16:03.734251 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:03.734226    2566 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-76cbf5c457-pzkvq: secret "image-registry-tls" not found
Jul 24 20:16:03.734251 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:03.734212    2566 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Jul 24 20:16:03.734369 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:03.734273    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls podName:58dc6394-715b-4b2a-a69f-29f9e21c81d9 nodeName:}" failed. No retries permitted until 2026-07-24 20:16:07.734259363 +0000 UTC m=+42.660398246 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls") pod "image-registry-76cbf5c457-pzkvq" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9") : secret "image-registry-tls" not found
Jul 24 20:16:03.734369 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:03.734286    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls podName:df2d6053-ad84-46c0-8cf6-60497f26536f nodeName:}" failed. No retries permitted until 2026-07-24 20:16:07.734280079 +0000 UTC m=+42.660418961 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls") pod "dns-default-7wtlx" (UID: "df2d6053-ad84-46c0-8cf6-60497f26536f") : secret "dns-default-metrics-tls" not found
Jul 24 20:16:03.884771 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:03.884686    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-2bds9" event={"ID":"da0cd617-b89e-4636-bc16-eee42b32327b","Type":"ContainerStarted","Data":"5df54e32104b89aeee9833f0be9e52c62577db9851c268816b03b79148b09a40"}
Jul 24 20:16:03.888070 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:03.888040    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-2pz67" event={"ID":"8efc8d87-bd58-4611-b380-47f8ddf39e81","Type":"ContainerStarted","Data":"98af5d5daa7b5b4800797ffc6b510ace044daf5a134e908719c3f70863596693"}
Jul 24 20:16:03.903486 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:03.903446    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-2bds9" podStartSLOduration=2.6173239820000003 podStartE2EDuration="4.903432069s" totalImagesPullingTime="2.286108087s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:03.903046142 +0000 UTC m=+38.829185047" watchObservedRunningTime="2026-07-24 20:16:03.903432069 +0000 UTC m=+38.829570973"
Jul 24 20:16:03.927500 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:03.927460    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-2pz67" podStartSLOduration=9.131247889 podStartE2EDuration="37.927448156s" totalImagesPullingTime="28.796200267s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:03.92690372 +0000 UTC m=+38.853042623" watchObservedRunningTime="2026-07-24 20:16:03.927448156 +0000 UTC m=+38.853587059"
Jul 24 20:16:07.762952 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.762919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:07.763422 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.762978    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:07.766231 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.766208    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/df2d6053-ad84-46c0-8cf6-60497f26536f-metrics-tls\") pod \"dns-default-7wtlx\" (UID: \"df2d6053-ad84-46c0-8cf6-60497f26536f\") " pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:07.766332 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.766296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"image-registry-76cbf5c457-pzkvq\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") " pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:07.794184 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.794157    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:07.815305 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.815282    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:07.950074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.950041    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-76cbf5c457-pzkvq"]
Jul 24 20:16:07.953771 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:07.953737    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58dc6394_715b_4b2a_a69f_29f9e21c81d9.slice/crio-ded15fdbff63acafad2159b5b2710a0e08fac0ea4408bb55507245ecb990683a WatchSource:0}: Error finding container ded15fdbff63acafad2159b5b2710a0e08fac0ea4408bb55507245ecb990683a: Status 404 returned error can't find the container with id ded15fdbff63acafad2159b5b2710a0e08fac0ea4408bb55507245ecb990683a
Jul 24 20:16:07.986860 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:07.986837    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-7wtlx"]
Jul 24 20:16:07.995394 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:07.995373    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf2d6053_ad84_46c0_8cf6_60497f26536f.slice/crio-8f7ef4d2129144e744d8531005cd04e4c0879abcd79b4efbf582357cf54fbc80 WatchSource:0}: Error finding container 8f7ef4d2129144e744d8531005cd04e4c0879abcd79b4efbf582357cf54fbc80: Status 404 returned error can't find the container with id 8f7ef4d2129144e744d8531005cd04e4c0879abcd79b4efbf582357cf54fbc80
Jul 24 20:16:08.902432 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:08.902391    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7wtlx" event={"ID":"df2d6053-ad84-46c0-8cf6-60497f26536f","Type":"ContainerStarted","Data":"8f7ef4d2129144e744d8531005cd04e4c0879abcd79b4efbf582357cf54fbc80"}
Jul 24 20:16:08.904143 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:08.904111    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq" event={"ID":"58dc6394-715b-4b2a-a69f-29f9e21c81d9","Type":"ContainerStarted","Data":"365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d"}
Jul 24 20:16:08.904271 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:08.904159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq" event={"ID":"58dc6394-715b-4b2a-a69f-29f9e21c81d9","Type":"ContainerStarted","Data":"ded15fdbff63acafad2159b5b2710a0e08fac0ea4408bb55507245ecb990683a"}
Jul 24 20:16:08.904271 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:08.904196    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:08.926605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:08.926545    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq" podStartSLOduration=19.926510919000002 podStartE2EDuration="19.926510919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:15:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:08.924421748 +0000 UTC m=+43.850560662" watchObservedRunningTime="2026-07-24 20:16:08.926510919 +0000 UTC m=+43.852649824"
Jul 24 20:16:10.909976 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:10.909940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7wtlx" event={"ID":"df2d6053-ad84-46c0-8cf6-60497f26536f","Type":"ContainerStarted","Data":"9d2351983c0c7a00063a10a960ed50a77220d555520f6ffea449bde698290dc2"}
Jul 24 20:16:10.910398 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:10.910018    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-7wtlx" event={"ID":"df2d6053-ad84-46c0-8cf6-60497f26536f","Type":"ContainerStarted","Data":"4e8e0ce2c1d1bfdfe282688b77f4089e3b10517300fd976fcc2d60f77d084a22"}
Jul 24 20:16:10.910398 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:10.910052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:20.914834 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:20.914799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-7wtlx"
Jul 24 20:16:20.936161 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:20.936107    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-7wtlx" podStartSLOduration=19.984492384 podStartE2EDuration="21.936092507s" totalImagesPullingTime="1.951600123s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:15:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:10.928785147 +0000 UTC m=+45.854924050" watchObservedRunningTime="2026-07-24 20:16:20.936092507 +0000 UTC m=+55.862231420"
Jul 24 20:16:24.495859 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.495825    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"]
Jul 24 20:16:24.530587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.530553    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"]
Jul 24 20:16:24.530834 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.530636    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.533513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.533488    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Jul 24 20:16:24.533513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.533490    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Jul 24 20:16:24.533707 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.533596    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Jul 24 20:16:24.534604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.534577    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Jul 24 20:16:24.534716 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.534643    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-7ptj6\""
Jul 24 20:16:24.544761 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.544740    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"]
Jul 24 20:16:24.544887 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.544871    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.547638 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.547617    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Jul 24 20:16:24.547787 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.547662    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Jul 24 20:16:24.547787 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.547725    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Jul 24 20:16:24.547906 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.547886    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Jul 24 20:16:24.560301 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.560282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"]
Jul 24 20:16:24.560386 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.560307    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"]
Jul 24 20:16:24.560386 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.560324    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"]
Jul 24 20:16:24.560386 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.560376    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.563576 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.563556    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Jul 24 20:16:24.569832 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.569813    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76cbf5c457-pzkvq"]
Jul 24 20:16:24.574021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.573997    2566 patch_prober.go:28] interesting pod/image-registry-76cbf5c457-pzkvq container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Jul 24 20:16:24.574128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.574048    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq" podUID="58dc6394-715b-4b2a-a69f-29f9e21c81d9" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 20:16:24.584037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584014    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-hub\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.584136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584044    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t8t8w\" (UniqueName: \"kubernetes.io/projected/77f3d860-19b4-4615-8da2-26986dea36cd-kube-api-access-t8t8w\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.584136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhmlb\" (UniqueName: \"kubernetes.io/projected/94d9fa33-6620-4039-9ade-523fcffb8b3c-kube-api-access-bhmlb\") pod \"managed-serviceaccount-addon-agent-54f9c65c8-47l7m\" (UID: \"94d9fa33-6620-4039-9ade-523fcffb8b3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.584225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94d9fa33-6620-4039-9ade-523fcffb8b3c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54f9c65c8-47l7m\" (UID: \"94d9fa33-6620-4039-9ade-523fcffb8b3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.584225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584173    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.584225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584198    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f3d860-19b4-4615-8da2-26986dea36cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.584331 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584229    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-ca\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.584331 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.584258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.595466 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.595441    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-c786m"]
Jul 24 20:16:24.614899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.614875    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.617520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.617498    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Jul 24 20:16:24.617743 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.617724    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Jul 24 20:16:24.618105 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.618086    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-5d6mv\""
Jul 24 20:16:24.618281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.618096    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Jul 24 20:16:24.618504 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.618482    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Jul 24 20:16:24.618728 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.618710    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c786m"]
Jul 24 20:16:24.685471 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tcz\" (UniqueName: \"kubernetes.io/projected/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-kube-api-access-d2tcz\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.685471 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685473    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-ca\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.685671 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685493    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.685671 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685602    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-hub\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.685671 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685640    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t8t8w\" (UniqueName: \"kubernetes.io/projected/77f3d860-19b4-4615-8da2-26986dea36cd-kube-api-access-t8t8w\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.685781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685677    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-data-volume\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.685781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685694    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94d9fa33-6620-4039-9ade-523fcffb8b3c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54f9c65c8-47l7m\" (UID: \"94d9fa33-6620-4039-9ade-523fcffb8b3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.685781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-tmp\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.685781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685737    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9mc6\" (UniqueName: \"kubernetes.io/projected/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-kube-api-access-f9mc6\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.686023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhmlb\" (UniqueName: \"kubernetes.io/projected/94d9fa33-6620-4039-9ade-523fcffb8b3c-kube-api-access-bhmlb\") pod \"managed-serviceaccount-addon-agent-54f9c65c8-47l7m\" (UID: \"94d9fa33-6620-4039-9ade-523fcffb8b3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.686023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685902    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-crio-socket\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.686023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685938    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f3d860-19b4-4615-8da2-26986dea36cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.686023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.685976    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.686023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.686007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.686275 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.686047    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.686275 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.686094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.686801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.686780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/77f3d860-19b4-4615-8da2-26986dea36cd-additional-service-ca\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.688253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.688231    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-ca\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.688336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.688281    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/94d9fa33-6620-4039-9ade-523fcffb8b3c-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-54f9c65c8-47l7m\" (UID: \"94d9fa33-6620-4039-9ade-523fcffb8b3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.688501 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.688480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-hub\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.688583 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.688568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.688893 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.688876    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/77f3d860-19b4-4615-8da2-26986dea36cd-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.697904 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.697879    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t8t8w\" (UniqueName: \"kubernetes.io/projected/77f3d860-19b4-4615-8da2-26986dea36cd-kube-api-access-t8t8w\") pod \"cluster-proxy-proxy-agent-76f459c6cb-rx8dm\" (UID: \"77f3d860-19b4-4615-8da2-26986dea36cd\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.698028 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.698012    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhmlb\" (UniqueName: \"kubernetes.io/projected/94d9fa33-6620-4039-9ade-523fcffb8b3c-kube-api-access-bhmlb\") pod \"managed-serviceaccount-addon-agent-54f9c65c8-47l7m\" (UID: \"94d9fa33-6620-4039-9ade-523fcffb8b3c\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.699621 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.699601    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-5d66954696-sl6sp"]
Jul 24 20:16:24.718580 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.718556    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.724888 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.724867    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d66954696-sl6sp"]
Jul 24 20:16:24.787115 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787041    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02220d65-aa06-4bf0-8e18-5ecb31743658-image-registry-private-configuration\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787115 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787081    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tcz\" (UniqueName: \"kubernetes.io/projected/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-kube-api-access-d2tcz\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.787300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787133    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-data-volume\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.787300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787166    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-tmp\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.787300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787182    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9mc6\" (UniqueName: \"kubernetes.io/projected/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-kube-api-access-f9mc6\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.787300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787210    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02220d65-aa06-4bf0-8e18-5ecb31743658-registry-certificates\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787233    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-crio-socket\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.787300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787283    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02220d65-aa06-4bf0-8e18-5ecb31743658-trusted-ca\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787373    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-bound-sa-token\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787402    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-crio-socket\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.787547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787411    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9js8\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-kube-api-access-k9js8\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787453    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.787547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787477    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-registry-tls\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787495    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-data-volume\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.787853 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-tmp\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.787853 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02220d65-aa06-4bf0-8e18-5ecb31743658-installation-pull-secrets\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787853 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787624    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.787853 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787641    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02220d65-aa06-4bf0-8e18-5ecb31743658-ca-trust-extracted\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.787853 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.787670    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.788097 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.788079    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.789690 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.789672    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.789781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.789767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-klusterlet-config\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.812616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.812590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tcz\" (UniqueName: \"kubernetes.io/projected/eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0-kube-api-access-d2tcz\") pod \"insights-runtime-extractor-c786m\" (UID: \"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0\") " pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:24.814504 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.814488    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9mc6\" (UniqueName: \"kubernetes.io/projected/fd42a228-c1f1-4d9b-a0d6-988e78772f1a-kube-api-access-f9mc6\") pod \"klusterlet-addon-workmgr-6b65777c7d-l97mg\" (UID: \"fd42a228-c1f1-4d9b-a0d6-988e78772f1a\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.852348 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.852318    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"
Jul 24 20:16:24.858545 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.858507    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"
Jul 24 20:16:24.877231 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.877201    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:24.888161 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888129    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-registry-tls\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02220d65-aa06-4bf0-8e18-5ecb31743658-installation-pull-secrets\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888197    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02220d65-aa06-4bf0-8e18-5ecb31743658-ca-trust-extracted\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888369 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02220d65-aa06-4bf0-8e18-5ecb31743658-image-registry-private-configuration\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888430 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02220d65-aa06-4bf0-8e18-5ecb31743658-registry-certificates\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888481 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888426    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02220d65-aa06-4bf0-8e18-5ecb31743658-trusted-ca\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888481 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888462    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-bound-sa-token\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888602 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888488    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9js8\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-kube-api-access-k9js8\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.888602 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.888572    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/02220d65-aa06-4bf0-8e18-5ecb31743658-ca-trust-extracted\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.889447 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.889419    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/02220d65-aa06-4bf0-8e18-5ecb31743658-registry-certificates\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.889612 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.889471    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/02220d65-aa06-4bf0-8e18-5ecb31743658-trusted-ca\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.890864 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.890841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/02220d65-aa06-4bf0-8e18-5ecb31743658-installation-pull-secrets\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.890965 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.890951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/02220d65-aa06-4bf0-8e18-5ecb31743658-image-registry-private-configuration\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.891092 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.891069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-registry-tls\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.907009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.906985    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-bound-sa-token\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.907472 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.907449    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9js8\" (UniqueName: \"kubernetes.io/projected/02220d65-aa06-4bf0-8e18-5ecb31743658-kube-api-access-k9js8\") pod \"image-registry-5d66954696-sl6sp\" (UID: \"02220d65-aa06-4bf0-8e18-5ecb31743658\") " pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:24.923606 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:24.923581    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-c786m"
Jul 24 20:16:25.015424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.015357    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m"]
Jul 24 20:16:25.017108 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:25.017055    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94d9fa33_6620_4039_9ade_523fcffb8b3c.slice/crio-2b9d825db6b162f07096ba8fd4c2d9137ced803faf655fd8afd6b0d5d21f2a10 WatchSource:0}: Error finding container 2b9d825db6b162f07096ba8fd4c2d9137ced803faf655fd8afd6b0d5d21f2a10: Status 404 returned error can't find the container with id 2b9d825db6b162f07096ba8fd4c2d9137ced803faf655fd8afd6b0d5d21f2a10
Jul 24 20:16:25.027741 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.026808    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:25.033625 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.033600    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm"]
Jul 24 20:16:25.036133 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:25.036104    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77f3d860_19b4_4615_8da2_26986dea36cd.slice/crio-084569d4d8068f57a6e49d1007d0ebfb8ac3c71896253e70baeabc5fefc14d01 WatchSource:0}: Error finding container 084569d4d8068f57a6e49d1007d0ebfb8ac3c71896253e70baeabc5fefc14d01: Status 404 returned error can't find the container with id 084569d4d8068f57a6e49d1007d0ebfb8ac3c71896253e70baeabc5fefc14d01
Jul 24 20:16:25.079659 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.079623    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"]
Jul 24 20:16:25.082709 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:25.082678    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd42a228_c1f1_4d9b_a0d6_988e78772f1a.slice/crio-10282d05febc178d24c3bdfae272c8f1eca9f8e0dceeb29042652d881ffc8235 WatchSource:0}: Error finding container 10282d05febc178d24c3bdfae272c8f1eca9f8e0dceeb29042652d881ffc8235: Status 404 returned error can't find the container with id 10282d05febc178d24c3bdfae272c8f1eca9f8e0dceeb29042652d881ffc8235
Jul 24 20:16:25.096093 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.096065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-c786m"]
Jul 24 20:16:25.099155 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:25.099037    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaa88a1d_58d6_4cf1_af2d_e81b74c4d9b0.slice/crio-fe5bef13cfb09444de823cf0d8f7fa141ac7752b225fd4b87ec3e3269cdf3e34 WatchSource:0}: Error finding container fe5bef13cfb09444de823cf0d8f7fa141ac7752b225fd4b87ec3e3269cdf3e34: Status 404 returned error can't find the container with id fe5bef13cfb09444de823cf0d8f7fa141ac7752b225fd4b87ec3e3269cdf3e34
Jul 24 20:16:25.171204 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.171180    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-5d66954696-sl6sp"]
Jul 24 20:16:25.173352 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:25.173328    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02220d65_aa06_4bf0_8e18_5ecb31743658.slice/crio-f98fa6a6391ca585c261a3dca415c33f14344d792ac0a0a7efaaec22e5ddb705 WatchSource:0}: Error finding container f98fa6a6391ca585c261a3dca415c33f14344d792ac0a0a7efaaec22e5ddb705: Status 404 returned error can't find the container with id f98fa6a6391ca585c261a3dca415c33f14344d792ac0a0a7efaaec22e5ddb705
Jul 24 20:16:25.875439 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.875233    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-5txqj"
Jul 24 20:16:25.951542 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.951493    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm" event={"ID":"77f3d860-19b4-4615-8da2-26986dea36cd","Type":"ContainerStarted","Data":"084569d4d8068f57a6e49d1007d0ebfb8ac3c71896253e70baeabc5fefc14d01"}
Jul 24 20:16:25.953609 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.953488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m" event={"ID":"94d9fa33-6620-4039-9ade-523fcffb8b3c","Type":"ContainerStarted","Data":"2b9d825db6b162f07096ba8fd4c2d9137ced803faf655fd8afd6b0d5d21f2a10"}
Jul 24 20:16:25.958781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.958738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d66954696-sl6sp" event={"ID":"02220d65-aa06-4bf0-8e18-5ecb31743658","Type":"ContainerStarted","Data":"a297452df249baf19c26e0980eeb12cdf0b8b4ac91c988188e8527a3b5c1b766"}
Jul 24 20:16:25.958910 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.958891    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-5d66954696-sl6sp" event={"ID":"02220d65-aa06-4bf0-8e18-5ecb31743658","Type":"ContainerStarted","Data":"f98fa6a6391ca585c261a3dca415c33f14344d792ac0a0a7efaaec22e5ddb705"}
Jul 24 20:16:25.959493 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.959469    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:25.962318 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.962237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c786m" event={"ID":"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0","Type":"ContainerStarted","Data":"fe5bef13cfb09444de823cf0d8f7fa141ac7752b225fd4b87ec3e3269cdf3e34"}
Jul 24 20:16:25.966652 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.966593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg" event={"ID":"fd42a228-c1f1-4d9b-a0d6-988e78772f1a","Type":"ContainerStarted","Data":"10282d05febc178d24c3bdfae272c8f1eca9f8e0dceeb29042652d881ffc8235"}
Jul 24 20:16:25.988160 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:25.987494    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-5d66954696-sl6sp" podStartSLOduration=1.987474779 podStartE2EDuration="1.987474779s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:16:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:25.98445155 +0000 UTC m=+60.910590454" watchObservedRunningTime="2026-07-24 20:16:25.987474779 +0000 UTC m=+60.913613683"
Jul 24 20:16:31.255136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.255094    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:16:31.258156 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.258134    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Jul 24 20:16:31.268370 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.268344    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Jul 24 20:16:31.278274 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.278249    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfpb\" (UniqueName: \"kubernetes.io/projected/d1eb465f-2b0c-400e-86c9-bf527b76e75a-kube-api-access-gcfpb\") pod \"network-check-target-bls72\" (UID: \"d1eb465f-2b0c-400e-86c9-bf527b76e75a\") " pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:16:31.356294 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.356255    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:16:31.356476 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.356329    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:16:31.359305 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.359277    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Jul 24 20:16:31.359427 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.359281    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Jul 24 20:16:31.369706 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.369682    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/01d20d74-1284-4562-aaac-c289b038f9b9-original-pull-secret\") pod \"global-pull-secret-syncer-gqxld\" (UID: \"01d20d74-1284-4562-aaac-c289b038f9b9\") " pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:16:31.369825 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.369709    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/95607563-c0dd-46eb-a0cf-c1d1ec957617-metrics-certs\") pod \"network-metrics-daemon-qh4xx\" (UID: \"95607563-c0dd-46eb-a0cf-c1d1ec957617\") " pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:16:31.399659 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.399622    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-gqxld"
Jul 24 20:16:31.409370 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.409349    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2sm2r\""
Jul 24 20:16:31.414955 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.414935    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-b8xdt\""
Jul 24 20:16:31.416836 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.416816    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-qh4xx"
Jul 24 20:16:31.423335 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:31.423317    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:16:34.574000 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:34.573928    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:34.763853 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:34.763171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-bls72"]
Jul 24 20:16:34.766601 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:34.766567    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1eb465f_2b0c_400e_86c9_bf527b76e75a.slice/crio-b53915d77ba78981df779a338291c7163fa1d35c55fdb4b75651770a779d1020 WatchSource:0}: Error finding container b53915d77ba78981df779a338291c7163fa1d35c55fdb4b75651770a779d1020: Status 404 returned error can't find the container with id b53915d77ba78981df779a338291c7163fa1d35c55fdb4b75651770a779d1020
Jul 24 20:16:34.813502 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:34.813412    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-qh4xx"]
Jul 24 20:16:34.818691 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:34.818665    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95607563_c0dd_46eb_a0cf_c1d1ec957617.slice/crio-106e7f4d1c1e7ea4a901a12af7c3ceffd6ed11c6809288d2f4b7787f6a6ebf0e WatchSource:0}: Error finding container 106e7f4d1c1e7ea4a901a12af7c3ceffd6ed11c6809288d2f4b7787f6a6ebf0e: Status 404 returned error can't find the container with id 106e7f4d1c1e7ea4a901a12af7c3ceffd6ed11c6809288d2f4b7787f6a6ebf0e
Jul 24 20:16:34.823146 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:34.823087    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-gqxld"]
Jul 24 20:16:34.827391 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:34.827349    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01d20d74_1284_4562_aaac_c289b038f9b9.slice/crio-6206584b6856c96c40b05f3505a41052a87359aa4d21a6c31bc786d6aa759f61 WatchSource:0}: Error finding container 6206584b6856c96c40b05f3505a41052a87359aa4d21a6c31bc786d6aa759f61: Status 404 returned error can't find the container with id 6206584b6856c96c40b05f3505a41052a87359aa4d21a6c31bc786d6aa759f61
Jul 24 20:16:34.999716 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:34.999675    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c786m" event={"ID":"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0","Type":"ContainerStarted","Data":"aff9bd94dfe60eb91e1f4c30ca322e198286eed78d243096f3f6e389a3c965f4"}
Jul 24 20:16:35.001068 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.001025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg" event={"ID":"fd42a228-c1f1-4d9b-a0d6-988e78772f1a","Type":"ContainerStarted","Data":"4c36d9bde9eac52d2f92923d630f47b491db0c14c7bb1fe42ec50bbb0e395ead"}
Jul 24 20:16:35.001225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.001197    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:35.002268 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.002243    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqxld" event={"ID":"01d20d74-1284-4562-aaac-c289b038f9b9","Type":"ContainerStarted","Data":"6206584b6856c96c40b05f3505a41052a87359aa4d21a6c31bc786d6aa759f61"}
Jul 24 20:16:35.003150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.003132    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg"
Jul 24 20:16:35.003509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.003490    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qh4xx" event={"ID":"95607563-c0dd-46eb-a0cf-c1d1ec957617","Type":"ContainerStarted","Data":"106e7f4d1c1e7ea4a901a12af7c3ceffd6ed11c6809288d2f4b7787f6a6ebf0e"}
Jul 24 20:16:35.005375 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.005353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm" event={"ID":"77f3d860-19b4-4615-8da2-26986dea36cd","Type":"ContainerStarted","Data":"2dfef6a935aa59cdc50295dd8567c05478690b8ffaf573aec8b5ebbca7e6cc61"}
Jul 24 20:16:35.005468 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.005387    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm" event={"ID":"77f3d860-19b4-4615-8da2-26986dea36cd","Type":"ContainerStarted","Data":"24fd3cbce84d443049b5b214550073f32e9a7d0a44b48276532125f86c7aaacc"}
Jul 24 20:16:35.005468 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.005407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm" event={"ID":"77f3d860-19b4-4615-8da2-26986dea36cd","Type":"ContainerStarted","Data":"694f7b41f831cc213b1f99cca3d860018bd6b5b07750ee222567c186d98a9dfc"}
Jul 24 20:16:35.006375 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.006354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bls72" event={"ID":"d1eb465f-2b0c-400e-86c9-bf527b76e75a","Type":"ContainerStarted","Data":"b53915d77ba78981df779a338291c7163fa1d35c55fdb4b75651770a779d1020"}
Jul 24 20:16:35.007505 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.007486    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m" event={"ID":"94d9fa33-6620-4039-9ade-523fcffb8b3c","Type":"ContainerStarted","Data":"70f1c4922175246047cc5c384e003a0df700ecbb65dd9a1c8f7f6621264cc83b"}
Jul 24 20:16:35.016693 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.016652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-6b65777c7d-l97mg" podStartSLOduration=1.510488453 podStartE2EDuration="11.016639266s" totalImagesPullingTime="9.506150813s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:35.016294816 +0000 UTC m=+69.942433720" watchObservedRunningTime="2026-07-24 20:16:35.016639266 +0000 UTC m=+69.942778164"
Jul 24 20:16:35.031692 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.031652    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-54f9c65c8-47l7m" podStartSLOduration=1.46572893 podStartE2EDuration="11.031640314s" totalImagesPullingTime="9.565911384s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:35.030457834 +0000 UTC m=+69.956596737" watchObservedRunningTime="2026-07-24 20:16:35.031640314 +0000 UTC m=+69.957779196"
Jul 24 20:16:35.049610 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:35.049560    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-76f459c6cb-rx8dm" podStartSLOduration=1.4963025779999999 podStartE2EDuration="11.049545546s" totalImagesPullingTime="9.553242968s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:16:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:35.04930334 +0000 UTC m=+69.975442246" watchObservedRunningTime="2026-07-24 20:16:35.049545546 +0000 UTC m=+69.975684441"
Jul 24 20:16:37.021095 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:37.021025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qh4xx" event={"ID":"95607563-c0dd-46eb-a0cf-c1d1ec957617","Type":"ContainerStarted","Data":"0b6613f67eae59fef12f7433c44555c5ce51ba58369255d232ce0d7c2d7a7ecb"}
Jul 24 20:16:37.021095 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:37.021098    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-qh4xx" event={"ID":"95607563-c0dd-46eb-a0cf-c1d1ec957617","Type":"ContainerStarted","Data":"fc2cfc448d2b403dc1b96e1e790c08df6eeef90123abd6bd4bb4dbb1e45321a9"}
Jul 24 20:16:37.045673 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:37.045586    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-qh4xx" podStartSLOduration=69.604398172 podStartE2EDuration="1m11.045567538s" totalImagesPullingTime="1.441169366s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:37.043408952 +0000 UTC m=+71.969547877" watchObservedRunningTime="2026-07-24 20:16:37.045567538 +0000 UTC m=+71.971706444"
Jul 24 20:16:39.350837 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.350802    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-zksdl"]
Jul 24 20:16:39.353593 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.353564    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.357314 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.357292    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Jul 24 20:16:39.358358 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.358324    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Jul 24 20:16:39.358358 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.358327    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Jul 24 20:16:39.363248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.363227    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Jul 24 20:16:39.363889 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.363778    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Jul 24 20:16:39.363889 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.363855    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Jul 24 20:16:39.364121 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.364100    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-p7mnv\""
Jul 24 20:16:39.426597 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426555    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-sys\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.426781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426612    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-textfile\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.426781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426636    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.426781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426673    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39765225-1850-48b8-bfb8-0f0396128605-metrics-client-ca\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.426781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426715    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-root\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.426781 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426770    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-tls\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.427022 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426831    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-accelerators-collector-config\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.427022 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426884    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z6xp2\" (UniqueName: \"kubernetes.io/projected/39765225-1850-48b8-bfb8-0f0396128605-kube-api-access-z6xp2\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.427022 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.426909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-wtmp\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527747 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527712    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39765225-1850-48b8-bfb8-0f0396128605-metrics-client-ca\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527754    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-root\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527782    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-tls\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527817    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-accelerators-collector-config\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527851    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z6xp2\" (UniqueName: \"kubernetes.io/projected/39765225-1850-48b8-bfb8-0f0396128605-kube-api-access-z6xp2\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-wtmp\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527913    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-root\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.527947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-sys\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.528307 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527969    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-textfile\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.528307 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.527994    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.528307 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.528086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-wtmp\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.528307 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.528142    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/39765225-1850-48b8-bfb8-0f0396128605-sys\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.528491 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.528394    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-textfile\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.528491 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.528423    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/39765225-1850-48b8-bfb8-0f0396128605-metrics-client-ca\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.528491 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.528454    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-accelerators-collector-config\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.530488 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.530460    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.530841 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.530821    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/39765225-1850-48b8-bfb8-0f0396128605-node-exporter-tls\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.538019 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.537993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z6xp2\" (UniqueName: \"kubernetes.io/projected/39765225-1850-48b8-bfb8-0f0396128605-kube-api-access-z6xp2\") pod \"node-exporter-zksdl\" (UID: \"39765225-1850-48b8-bfb8-0f0396128605\") " pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:39.665473 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:39.665386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-zksdl"
Jul 24 20:16:41.260154 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:16:41.260126    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39765225_1850_48b8_bfb8_0f0396128605.slice/crio-e46981c773247ecaa8d5f228193f573e5a8cabebad2cfeb3b02ddc35d0082a2b WatchSource:0}: Error finding container e46981c773247ecaa8d5f228193f573e5a8cabebad2cfeb3b02ddc35d0082a2b: Status 404 returned error can't find the container with id e46981c773247ecaa8d5f228193f573e5a8cabebad2cfeb3b02ddc35d0082a2b
Jul 24 20:16:42.041785 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:42.041693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-bls72" event={"ID":"d1eb465f-2b0c-400e-86c9-bf527b76e75a","Type":"ContainerStarted","Data":"97a984ab03b76a3e86b9e71033825666b356126c69e9ddd71e83d15558e10aa7"}
Jul 24 20:16:42.041950 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:42.041856    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:16:42.043670 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:42.043643    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c786m" event={"ID":"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0","Type":"ContainerStarted","Data":"de4287d32b1c42c63d88114ae52fe35b27fde1ee4fcc4c059c13edf6f22395d6"}
Jul 24 20:16:42.045350 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:42.045319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-gqxld" event={"ID":"01d20d74-1284-4562-aaac-c289b038f9b9","Type":"ContainerStarted","Data":"81f074fe569a38857c7e0ad93243af7fdf1481cbd106fe2f7ef63007560682a9"}
Jul 24 20:16:42.046925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:42.046886    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zksdl" event={"ID":"39765225-1850-48b8-bfb8-0f0396128605","Type":"ContainerStarted","Data":"e46981c773247ecaa8d5f228193f573e5a8cabebad2cfeb3b02ddc35d0082a2b"}
Jul 24 20:16:42.059691 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:42.059626    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-bls72" podStartSLOduration=69.570876989 podStartE2EDuration="1m16.059603615s" totalImagesPullingTime="6.488726626s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:42.058747577 +0000 UTC m=+76.984886501" watchObservedRunningTime="2026-07-24 20:16:42.059603615 +0000 UTC m=+76.985742521"
Jul 24 20:16:42.079442 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:42.079370    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-gqxld" podStartSLOduration=69.651567895 podStartE2EDuration="1m16.079346914s" totalImagesPullingTime="6.427779019s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:15:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:42.078792937 +0000 UTC m=+77.004931843" watchObservedRunningTime="2026-07-24 20:16:42.079346914 +0000 UTC m=+77.005485816"
Jul 24 20:16:43.052478 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:43.052442    2566 generic.go:356] "Generic (PLEG): container finished" podID="39765225-1850-48b8-bfb8-0f0396128605" containerID="72f10634f2f84819ade94b08d2d976fc91c732daf68d238c60da6fc227b3f987" exitCode=0
Jul 24 20:16:43.052935 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:43.052549    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zksdl" event={"ID":"39765225-1850-48b8-bfb8-0f0396128605","Type":"ContainerDied","Data":"72f10634f2f84819ade94b08d2d976fc91c732daf68d238c60da6fc227b3f987"}
Jul 24 20:16:45.062067 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:45.061980    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zksdl" event={"ID":"39765225-1850-48b8-bfb8-0f0396128605","Type":"ContainerStarted","Data":"01cbf41a534487cd79faec3823b003983fcd46284d1a25625b995b81dcde61af"}
Jul 24 20:16:45.062067 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:45.062072    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-zksdl" event={"ID":"39765225-1850-48b8-bfb8-0f0396128605","Type":"ContainerStarted","Data":"b27752f88ae77081aab44587904193facc900c1013f4a02b4791d0827fdf97c9"}
Jul 24 20:16:45.063948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:45.063921    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-c786m" event={"ID":"eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0","Type":"ContainerStarted","Data":"dcd562581d213f1354ac6eb69d258afdb3f5de5760d72a5857e9cefb71165873"}
Jul 24 20:16:45.087874 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:45.087823    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-zksdl" podStartSLOduration=5.232985826 podStartE2EDuration="6.087808336s" totalImagesPullingTime="777.948131ms" totalInitContainerRuntime="76.874379ms" isStatefulPod=true podCreationTimestamp="2026-07-24 20:16:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:45.085738379 +0000 UTC m=+80.011877305" watchObservedRunningTime="2026-07-24 20:16:45.087808336 +0000 UTC m=+80.013947242"
Jul 24 20:16:46.978038 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:46.978006    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-5d66954696-sl6sp"
Jul 24 20:16:47.001580 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:47.001432    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-c786m" podStartSLOduration=3.497264659 podStartE2EDuration="23.001409436s" totalImagesPullingTime="19.504144777s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 20:16:24 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:16:45.113773542 +0000 UTC m=+80.039912448" watchObservedRunningTime="2026-07-24 20:16:47.001409436 +0000 UTC m=+81.927548342"
Jul 24 20:16:49.588344 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.588300    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq" podUID="58dc6394-715b-4b2a-a69f-29f9e21c81d9" containerName="registry" containerID="cri-o://365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d" gracePeriod=30
Jul 24 20:16:49.820441 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.820418    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:49.908399 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58dc6394-715b-4b2a-a69f-29f9e21c81d9-ca-trust-extracted\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908399 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908406    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-trusted-ca\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908683 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908427    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-image-registry-private-configuration\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908683 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908450    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-bound-sa-token\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908683 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908482    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908683 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908498    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-installation-pull-secrets\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908683 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908518    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcf5q\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-kube-api-access-dcf5q\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908683 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908557    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-certificates\") pod \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\" (UID: \"58dc6394-715b-4b2a-a69f-29f9e21c81d9\") "
Jul 24 20:16:49.908992 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.908856    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:16:49.909175 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.909146    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:16:49.911033 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.910978    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-kube-api-access-dcf5q" (OuterVolumeSpecName: "kube-api-access-dcf5q") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "kube-api-access-dcf5q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:16:49.911171 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.911083    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:16:49.911171 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.911118    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:16:49.911295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.911193    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:16:49.911295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.911233    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:16:49.917914 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:49.917891    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58dc6394-715b-4b2a-a69f-29f9e21c81d9-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "58dc6394-715b-4b2a-a69f-29f9e21c81d9" (UID: "58dc6394-715b-4b2a-a69f-29f9e21c81d9"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:16:50.009341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009310    2566 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/58dc6394-715b-4b2a-a69f-29f9e21c81d9-ca-trust-extracted\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.009341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009336    2566 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-trusted-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.009341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009347    2566 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-image-registry-private-configuration\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.009509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009356    2566 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-bound-sa-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.009509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009364    2566 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.009509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009372    2566 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/58dc6394-715b-4b2a-a69f-29f9e21c81d9-installation-pull-secrets\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.009509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009379    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcf5q\" (UniqueName: \"kubernetes.io/projected/58dc6394-715b-4b2a-a69f-29f9e21c81d9-kube-api-access-dcf5q\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.009509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.009388    2566 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/58dc6394-715b-4b2a-a69f-29f9e21c81d9-registry-certificates\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:16:50.084073 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.084043    2566 generic.go:356] "Generic (PLEG): container finished" podID="58dc6394-715b-4b2a-a69f-29f9e21c81d9" containerID="365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d" exitCode=0
Jul 24 20:16:50.084202 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.084087    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq"
Jul 24 20:16:50.084202 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.084083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq" event={"ID":"58dc6394-715b-4b2a-a69f-29f9e21c81d9","Type":"ContainerDied","Data":"365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d"}
Jul 24 20:16:50.084303 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.084211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-76cbf5c457-pzkvq" event={"ID":"58dc6394-715b-4b2a-a69f-29f9e21c81d9","Type":"ContainerDied","Data":"ded15fdbff63acafad2159b5b2710a0e08fac0ea4408bb55507245ecb990683a"}
Jul 24 20:16:50.084303 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.084236    2566 scope.go:117] "RemoveContainer" containerID="365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d"
Jul 24 20:16:50.092582 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.092567    2566 scope.go:117] "RemoveContainer" containerID="365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d"
Jul 24 20:16:50.092844 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:16:50.092823    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d\": container with ID starting with 365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d not found: ID does not exist" containerID="365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d"
Jul 24 20:16:50.092908 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.092860    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d"} err="failed to get container status \"365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d\": rpc error: code = NotFound desc = could not find container \"365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d\": container with ID starting with 365103893aa6785de2da6a7e65d5ce31df06c601080c7e52e1e581172210a52d not found: ID does not exist"
Jul 24 20:16:50.105706 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.105684    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-76cbf5c457-pzkvq"]
Jul 24 20:16:50.111281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:50.111262    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-76cbf5c457-pzkvq"]
Jul 24 20:16:51.691225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:16:51.691187    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="58dc6394-715b-4b2a-a69f-29f9e21c81d9" path="/var/lib/kubelet/pods/58dc6394-715b-4b2a-a69f-29f9e21c81d9/volumes"
Jul 24 20:17:13.055185 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:17:13.055153    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-bls72"
Jul 24 20:20:25.433794 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:20:25.433767    2566 kubelet.go:1648] "Image garbage collection succeeded"
Jul 24 20:21:10.334554 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.334508    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"]
Jul 24 20:21:10.334984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.334743    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="58dc6394-715b-4b2a-a69f-29f9e21c81d9" containerName="registry"
Jul 24 20:21:10.334984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.334754    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="58dc6394-715b-4b2a-a69f-29f9e21c81d9" containerName="registry"
Jul 24 20:21:10.334984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.334815    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="58dc6394-715b-4b2a-a69f-29f9e21c81d9" containerName="registry"
Jul 24 20:21:10.336449 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.336432    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.339408 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.339385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Jul 24 20:21:10.339547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.339408    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Jul 24 20:21:10.339725 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.339711    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-lmv5f\""
Jul 24 20:21:10.339988 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.339969    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Jul 24 20:21:10.340575 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.340559    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Jul 24 20:21:10.355777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.355756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"]
Jul 24 20:21:10.467473 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.467417    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e1fa0e0-2caf-453a-b41a-e69f105e783b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.467705 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.467585    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e1fa0e0-2caf-453a-b41a-e69f105e783b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.467705 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.467622    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh89k\" (UniqueName: \"kubernetes.io/projected/0e1fa0e0-2caf-453a-b41a-e69f105e783b-kube-api-access-qh89k\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.467705 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.467661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e1fa0e0-2caf-453a-b41a-e69f105e783b-tmp\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.467705 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.467694    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e1fa0e0-2caf-453a-b41a-e69f105e783b-manifests\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.568278 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.568218    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e1fa0e0-2caf-453a-b41a-e69f105e783b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.568278 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.568270    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh89k\" (UniqueName: \"kubernetes.io/projected/0e1fa0e0-2caf-453a-b41a-e69f105e783b-kube-api-access-qh89k\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.568574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.568296    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e1fa0e0-2caf-453a-b41a-e69f105e783b-tmp\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.568574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.568330    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e1fa0e0-2caf-453a-b41a-e69f105e783b-manifests\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.568574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.568366    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e1fa0e0-2caf-453a-b41a-e69f105e783b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.568871 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.568843    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/0e1fa0e0-2caf-453a-b41a-e69f105e783b-manifests\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.570707 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.570675    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/0e1fa0e0-2caf-453a-b41a-e69f105e783b-tmp\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.570906 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.570886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/0e1fa0e0-2caf-453a-b41a-e69f105e783b-apiservice-cert\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.570981 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.570887    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/0e1fa0e0-2caf-453a-b41a-e69f105e783b-webhook-cert\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.578836 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.578817    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh89k\" (UniqueName: \"kubernetes.io/projected/0e1fa0e0-2caf-453a-b41a-e69f105e783b-kube-api-access-qh89k\") pod \"opendatahub-operator-controller-manager-7c48cfcf9b-rf42p\" (UID: \"0e1fa0e0-2caf-453a-b41a-e69f105e783b\") " pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.646286 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.646255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:10.781196 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.781167    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"]
Jul 24 20:21:10.784704 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:10.784674    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e1fa0e0_2caf_453a_b41a_e69f105e783b.slice/crio-6a0b36985423e9662bacf41ca1d55a90bbddcb22fb49bd61ac3db4f4fb31701d WatchSource:0}: Error finding container 6a0b36985423e9662bacf41ca1d55a90bbddcb22fb49bd61ac3db4f4fb31701d: Status 404 returned error can't find the container with id 6a0b36985423e9662bacf41ca1d55a90bbddcb22fb49bd61ac3db4f4fb31701d
Jul 24 20:21:10.786304 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.786287    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 20:21:10.906844 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:10.906752    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p" event={"ID":"0e1fa0e0-2caf-453a-b41a-e69f105e783b","Type":"ContainerStarted","Data":"6a0b36985423e9662bacf41ca1d55a90bbddcb22fb49bd61ac3db4f4fb31701d"}
Jul 24 20:21:14.923181 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:14.923146    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p" event={"ID":"0e1fa0e0-2caf-453a-b41a-e69f105e783b","Type":"ContainerStarted","Data":"2d2a31d37a4fac454f9cb7207d7297cd409a1696f8e5f18b5a7df6a06290b45e"}
Jul 24 20:21:15.928654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:15.928612    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e1fa0e0-2caf-453a-b41a-e69f105e783b" containerID="2d2a31d37a4fac454f9cb7207d7297cd409a1696f8e5f18b5a7df6a06290b45e" exitCode=0
Jul 24 20:21:15.929096 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:15.928693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p" event={"ID":"0e1fa0e0-2caf-453a-b41a-e69f105e783b","Type":"ContainerDied","Data":"2d2a31d37a4fac454f9cb7207d7297cd409a1696f8e5f18b5a7df6a06290b45e"}
Jul 24 20:21:16.933943 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:16.933905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p" event={"ID":"0e1fa0e0-2caf-453a-b41a-e69f105e783b","Type":"ContainerStarted","Data":"7e38d1597ff089adb3818a9a7c92c50bb6003e665bc3de8fa3681d6fb6c30b2e"}
Jul 24 20:21:16.934419 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:16.934052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:16.957142 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:16.957084    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p" podStartSLOduration=2.414924518 podStartE2EDuration="6.957068228s" totalImagesPullingTime="3.464291074s" totalInitContainerRuntime="1.077852636s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:10 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:16.955153135 +0000 UTC m=+351.881292063" watchObservedRunningTime="2026-07-24 20:21:16.957068228 +0000 UTC m=+351.883207132"
Jul 24 20:21:27.939114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:27.939080    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-7c48cfcf9b-rf42p"
Jul 24 20:21:30.262640 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.262609    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"]
Jul 24 20:21:30.272274 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.272251    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.275456 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.275435    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Jul 24 20:21:30.275656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.275636    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Jul 24 20:21:30.275736 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.275640    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Jul 24 20:21:30.275871 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.275857    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Jul 24 20:21:30.283087 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.283065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"]
Jul 24 20:21:30.303302 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.303276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s4bb\" (UniqueName: \"kubernetes.io/projected/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-kube-api-access-9s4bb\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.303396 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.303343    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tls-certs\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.303396 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.303384    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tmp\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.404613 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.404570    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s4bb\" (UniqueName: \"kubernetes.io/projected/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-kube-api-access-9s4bb\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.404814 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.404628    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tls-certs\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.404814 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.404662    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tmp\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.404814 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:30.404797    2566 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Jul 24 20:21:30.404984 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:30.404873    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tls-certs podName:bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b nodeName:}" failed. No retries permitted until 2026-07-24 20:21:30.904850815 +0000 UTC m=+365.830989710 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tls-certs") pod "kube-auth-proxy-68fdc99b6-zff46" (UID: "bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b") : secret "kube-auth-proxy-tls" not found
Jul 24 20:21:30.406789 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.406772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tmp\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.428248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.428218    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s4bb\" (UniqueName: \"kubernetes.io/projected/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-kube-api-access-9s4bb\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.514925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.514869    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wqdt7"]
Jul 24 20:21:30.516884 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.516871    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:30.520003 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.519982    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-gcvws\""
Jul 24 20:21:30.520096 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.520080    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\""
Jul 24 20:21:30.533574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.533553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wqdt7"]
Jul 24 20:21:30.606414 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.606389    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fmbkq\" (UniqueName: \"kubernetes.io/projected/f702b75b-6e15-434a-92d1-c77d389d6e60-kube-api-access-fmbkq\") pod \"odh-model-controller-7998c67dc4-wqdt7\" (UID: \"f702b75b-6e15-434a-92d1-c77d389d6e60\") " pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:30.606537 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.606433    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f702b75b-6e15-434a-92d1-c77d389d6e60-cert\") pod \"odh-model-controller-7998c67dc4-wqdt7\" (UID: \"f702b75b-6e15-434a-92d1-c77d389d6e60\") " pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:30.706930 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.706905    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fmbkq\" (UniqueName: \"kubernetes.io/projected/f702b75b-6e15-434a-92d1-c77d389d6e60-kube-api-access-fmbkq\") pod \"odh-model-controller-7998c67dc4-wqdt7\" (UID: \"f702b75b-6e15-434a-92d1-c77d389d6e60\") " pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:30.707026 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.706944    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f702b75b-6e15-434a-92d1-c77d389d6e60-cert\") pod \"odh-model-controller-7998c67dc4-wqdt7\" (UID: \"f702b75b-6e15-434a-92d1-c77d389d6e60\") " pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:30.707075 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:30.707050    2566 secret.go:189] Couldn't get secret opendatahub/odh-model-controller-webhook-cert: secret "odh-model-controller-webhook-cert" not found
Jul 24 20:21:30.707136 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:30.707124    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f702b75b-6e15-434a-92d1-c77d389d6e60-cert podName:f702b75b-6e15-434a-92d1-c77d389d6e60 nodeName:}" failed. No retries permitted until 2026-07-24 20:21:31.207105214 +0000 UTC m=+366.133244100 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/f702b75b-6e15-434a-92d1-c77d389d6e60-cert") pod "odh-model-controller-7998c67dc4-wqdt7" (UID: "f702b75b-6e15-434a-92d1-c77d389d6e60") : secret "odh-model-controller-webhook-cert" not found
Jul 24 20:21:30.715400 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.715377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fmbkq\" (UniqueName: \"kubernetes.io/projected/f702b75b-6e15-434a-92d1-c77d389d6e60-kube-api-access-fmbkq\") pod \"odh-model-controller-7998c67dc4-wqdt7\" (UID: \"f702b75b-6e15-434a-92d1-c77d389d6e60\") " pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:30.908019 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.907981    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tls-certs\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:30.910312 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:30.910288    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b-tls-certs\") pod \"kube-auth-proxy-68fdc99b6-zff46\" (UID: \"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b\") " pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:31.181481 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.181396    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"
Jul 24 20:21:31.215645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.215615    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f702b75b-6e15-434a-92d1-c77d389d6e60-cert\") pod \"odh-model-controller-7998c67dc4-wqdt7\" (UID: \"f702b75b-6e15-434a-92d1-c77d389d6e60\") " pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:31.218164 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.218136    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/f702b75b-6e15-434a-92d1-c77d389d6e60-cert\") pod \"odh-model-controller-7998c67dc4-wqdt7\" (UID: \"f702b75b-6e15-434a-92d1-c77d389d6e60\") " pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:31.306979 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.306946    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-68fdc99b6-zff46"]
Jul 24 20:21:31.309583 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:31.309556    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc47ac3d_c6f6_4cdb_94ca_6cd0942d4f7b.slice/crio-8c35fc95077202356dba85d5c43bfc9e14cf536ee91a05dfb3bc0986842cb1c9 WatchSource:0}: Error finding container 8c35fc95077202356dba85d5c43bfc9e14cf536ee91a05dfb3bc0986842cb1c9: Status 404 returned error can't find the container with id 8c35fc95077202356dba85d5c43bfc9e14cf536ee91a05dfb3bc0986842cb1c9
Jul 24 20:21:31.426403 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.426377    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:31.513276 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.513236    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dtkrp"]
Jul 24 20:21:31.515670 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.515649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:31.520170 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.520151    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Jul 24 20:21:31.533438 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.533411    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dtkrp"]
Jul 24 20:21:31.578057 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.578036    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-7998c67dc4-wqdt7"]
Jul 24 20:21:31.581325 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:31.581304    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf702b75b_6e15_434a_92d1_c77d389d6e60.slice/crio-18f19b97bc1d68204b01c9fce74b5f8aefbeedd83c76ac7a91e69534156bb542 WatchSource:0}: Error finding container 18f19b97bc1d68204b01c9fce74b5f8aefbeedd83c76ac7a91e69534156bb542: Status 404 returned error can't find the container with id 18f19b97bc1d68204b01c9fce74b5f8aefbeedd83c76ac7a91e69534156bb542
Jul 24 20:21:31.619740 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.619717    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fa11de-e020-4f26-ab04-f0cadc0e4301-tls-certs\") pod \"model-serving-api-54d89bd5fc-dtkrp\" (UID: \"54fa11de-e020-4f26-ab04-f0cadc0e4301\") " pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:31.619827 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.619744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qllzp\" (UniqueName: \"kubernetes.io/projected/54fa11de-e020-4f26-ab04-f0cadc0e4301-kube-api-access-qllzp\") pod \"model-serving-api-54d89bd5fc-dtkrp\" (UID: \"54fa11de-e020-4f26-ab04-f0cadc0e4301\") " pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:31.720668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.720620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fa11de-e020-4f26-ab04-f0cadc0e4301-tls-certs\") pod \"model-serving-api-54d89bd5fc-dtkrp\" (UID: \"54fa11de-e020-4f26-ab04-f0cadc0e4301\") " pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:31.720668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.720648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qllzp\" (UniqueName: \"kubernetes.io/projected/54fa11de-e020-4f26-ab04-f0cadc0e4301-kube-api-access-qllzp\") pod \"model-serving-api-54d89bd5fc-dtkrp\" (UID: \"54fa11de-e020-4f26-ab04-f0cadc0e4301\") " pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:31.720782 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:31.720766    2566 secret.go:189] Couldn't get secret opendatahub/model-serving-api-tls: secret "model-serving-api-tls" not found
Jul 24 20:21:31.720836 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:31.720826    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/54fa11de-e020-4f26-ab04-f0cadc0e4301-tls-certs podName:54fa11de-e020-4f26-ab04-f0cadc0e4301 nodeName:}" failed. No retries permitted until 2026-07-24 20:21:32.220808416 +0000 UTC m=+367.146947314 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/54fa11de-e020-4f26-ab04-f0cadc0e4301-tls-certs") pod "model-serving-api-54d89bd5fc-dtkrp" (UID: "54fa11de-e020-4f26-ab04-f0cadc0e4301") : secret "model-serving-api-tls" not found
Jul 24 20:21:31.734367 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.734345    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qllzp\" (UniqueName: \"kubernetes.io/projected/54fa11de-e020-4f26-ab04-f0cadc0e4301-kube-api-access-qllzp\") pod \"model-serving-api-54d89bd5fc-dtkrp\" (UID: \"54fa11de-e020-4f26-ab04-f0cadc0e4301\") " pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:31.809564 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.809539    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"]
Jul 24 20:21:31.811496 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.811482    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.813758 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.813736    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Jul 24 20:21:31.813872 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.813821    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Jul 24 20:21:31.814159 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.814140    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-57ww5\""
Jul 24 20:21:31.814224 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.814169    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Jul 24 20:21:31.814294 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.814280    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Jul 24 20:21:31.830387 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.830367    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"]
Jul 24 20:21:31.922354 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.922328    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.922444 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.922369    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.922444 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.922398    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.922444 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.922439    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crgz8\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-kube-api-access-crgz8\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.922642 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.922538    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.922642 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.922568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.922642 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.922604    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:31.980340 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.980283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46" event={"ID":"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b","Type":"ContainerStarted","Data":"8c35fc95077202356dba85d5c43bfc9e14cf536ee91a05dfb3bc0986842cb1c9"}
Jul 24 20:21:31.981810 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:31.981779    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7" event={"ID":"f702b75b-6e15-434a-92d1-c77d389d6e60","Type":"ContainerStarted","Data":"18f19b97bc1d68204b01c9fce74b5f8aefbeedd83c76ac7a91e69534156bb542"}
Jul 24 20:21:32.023242 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023215    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.023339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023254    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.023339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023291    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.023339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023335    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.023517 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023497    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.023604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023554    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.023604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crgz8\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-kube-api-access-crgz8\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.023836 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.023797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.025506 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.025489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.025673 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.025657    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.025985 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.025970    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.026026 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.025986    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.031919 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.031899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.032164 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.032143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-crgz8\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-kube-api-access-crgz8\") pod \"istiod-openshift-gateway-7b7995d5cd-vtc6n\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.120227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.120199    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:32.225084 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.225020    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fa11de-e020-4f26-ab04-f0cadc0e4301-tls-certs\") pod \"model-serving-api-54d89bd5fc-dtkrp\" (UID: \"54fa11de-e020-4f26-ab04-f0cadc0e4301\") " pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:32.227892 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.227863    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/54fa11de-e020-4f26-ab04-f0cadc0e4301-tls-certs\") pod \"model-serving-api-54d89bd5fc-dtkrp\" (UID: \"54fa11de-e020-4f26-ab04-f0cadc0e4301\") " pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:32.265090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.264909    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"]
Jul 24 20:21:32.267516 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:32.267482    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3b4886_da51_4d25_bd9f_c5a8846c9e93.slice/crio-fc14a8baf3690e393e585d1eb72740fd01b7a90d332a1b9cb74af6b20140cfef WatchSource:0}: Error finding container fc14a8baf3690e393e585d1eb72740fd01b7a90d332a1b9cb74af6b20140cfef: Status 404 returned error can't find the container with id fc14a8baf3690e393e585d1eb72740fd01b7a90d332a1b9cb74af6b20140cfef
Jul 24 20:21:32.426605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.426522    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:32.602108 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.602024    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-dtkrp"]
Jul 24 20:21:32.605763 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:32.605733    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fa11de_e020_4f26_ab04_f0cadc0e4301.slice/crio-9f94c377bf3cd08e7565d614df7288ddc1a54a720abe54f21dd2862ad923b1e5 WatchSource:0}: Error finding container 9f94c377bf3cd08e7565d614df7288ddc1a54a720abe54f21dd2862ad923b1e5: Status 404 returned error can't find the container with id 9f94c377bf3cd08e7565d614df7288ddc1a54a720abe54f21dd2862ad923b1e5
Jul 24 20:21:32.990427 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.990354    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n" event={"ID":"9b3b4886-da51-4d25-bd9f-c5a8846c9e93","Type":"ContainerStarted","Data":"fc14a8baf3690e393e585d1eb72740fd01b7a90d332a1b9cb74af6b20140cfef"}
Jul 24 20:21:32.992193 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:32.992165    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp" event={"ID":"54fa11de-e020-4f26-ab04-f0cadc0e4301","Type":"ContainerStarted","Data":"9f94c377bf3cd08e7565d614df7288ddc1a54a720abe54f21dd2862ad923b1e5"}
Jul 24 20:21:36.519318 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:36.519279    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:21:36.519664 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:36.519362    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:21:37.008981 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.008945    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n" event={"ID":"9b3b4886-da51-4d25-bd9f-c5a8846c9e93","Type":"ContainerStarted","Data":"89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d"}
Jul 24 20:21:37.009197 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.009139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:37.010805 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.010767    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7" event={"ID":"f702b75b-6e15-434a-92d1-c77d389d6e60","Type":"ContainerStarted","Data":"d81ce5659d341c5b3e43e9e7dcc64e1d0e325d88b0426c3c5ba5335c44bda0d1"}
Jul 24 20:21:37.010956 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.010855    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:37.011264 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.011239    2566 patch_prober.go:28] interesting pod/istiod-openshift-gateway-7b7995d5cd-vtc6n container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 20:21:37.011363 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.011288    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n" podUID="9b3b4886-da51-4d25-bd9f-c5a8846c9e93" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 20:21:37.012338 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.012315    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp" event={"ID":"54fa11de-e020-4f26-ab04-f0cadc0e4301","Type":"ContainerStarted","Data":"560f79bb051c6c945f8f01365563b9de5bd30e5512988000bd78439d5a23be52"}
Jul 24 20:21:37.012446 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.012404    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:21:37.013676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.013655    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46" event={"ID":"bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b","Type":"ContainerStarted","Data":"a6be4acb9730fb124d99d5f24426ab9b5e213550cd9c4322c6925410d9db6998"}
Jul 24 20:21:37.050823 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.050761    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7" podStartSLOduration=2.108463173 podStartE2EDuration="7.050745809s" totalImagesPullingTime="4.942282636s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:37.049471418 +0000 UTC m=+371.975610323" watchObservedRunningTime="2026-07-24 20:21:37.050745809 +0000 UTC m=+371.976884713"
Jul 24 20:21:37.051023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.050871    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n" podStartSLOduration=1.802124888 podStartE2EDuration="6.050863204s" totalImagesPullingTime="4.248738316s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:37.030438951 +0000 UTC m=+371.956577857" watchObservedRunningTime="2026-07-24 20:21:37.050863204 +0000 UTC m=+371.977002110"
Jul 24 20:21:37.082792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.082736    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-68fdc99b6-zff46" podStartSLOduration=1.8784091520000001 podStartE2EDuration="7.082718515s" totalImagesPullingTime="5.204309363s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:37.081624424 +0000 UTC m=+372.007763349" watchObservedRunningTime="2026-07-24 20:21:37.082718515 +0000 UTC m=+372.008857421"
Jul 24 20:21:37.110661 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.110589    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp" podStartSLOduration=2.20372409 podStartE2EDuration="6.110569562s" totalImagesPullingTime="3.906845472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:37.109852265 +0000 UTC m=+372.035991169" watchObservedRunningTime="2026-07-24 20:21:37.110569562 +0000 UTC m=+372.036708468"
Jul 24 20:21:37.923074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.923041    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"]
Jul 24 20:21:37.926040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.926016    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.929977 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.929952    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"data-science-gateway-data-science-gateway-class-dockercfg-k9zwx\""
Jul 24 20:21:37.958298 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.958263    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"]
Jul 24 20:21:37.969952 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.969915    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.969961    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.970023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8c6vh\" (UniqueName: \"kubernetes.io/projected/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-kube-api-access-8c6vh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.970066    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.970129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.970171    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.970196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.970223    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:37.970385 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:37.970258    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.018007 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.017980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:38.071010 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.070973    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.071168 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.071019    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.071249 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.071225    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8c6vh\" (UniqueName: \"kubernetes.io/projected/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-kube-api-access-8c6vh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.071298 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.071282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.071422 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.071391    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-data\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.071552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.071520    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.072930 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.072899    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istiod-ca-cert\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.073041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.072986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.073101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.073056    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.073157 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.073109    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.073208 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.073181    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.077074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.073285    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-credential-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.077074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.073612    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-workload-socket\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.077074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.073698    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-envoy\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.077742 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.077722    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-workload-certs\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.080340 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.080316    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-podinfo\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.082882 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.082858    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8c6vh\" (UniqueName: \"kubernetes.io/projected/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-kube-api-access-8c6vh\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.083879 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.083860    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2c88b65a-ebb3-4dc9-a51c-3d813671b8df-istio-token\") pod \"data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c\" (UID: \"2c88b65a-ebb3-4dc9-a51c-3d813671b8df\") " pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.235927 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.235837    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:38.394932 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:38.394899    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"]
Jul 24 20:21:38.396923 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:38.396891    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2c88b65a_ebb3_4dc9_a51c_3d813671b8df.slice/crio-549c81957bc4702bb36051234811420793ad1d0b4e951289ec8cae099ccf14c3 WatchSource:0}: Error finding container 549c81957bc4702bb36051234811420793ad1d0b4e951289ec8cae099ccf14c3: Status 404 returned error can't find the container with id 549c81957bc4702bb36051234811420793ad1d0b4e951289ec8cae099ccf14c3
Jul 24 20:21:39.022176 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:39.022140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c" event={"ID":"2c88b65a-ebb3-4dc9-a51c-3d813671b8df","Type":"ContainerStarted","Data":"549c81957bc4702bb36051234811420793ad1d0b4e951289ec8cae099ccf14c3"}
Jul 24 20:21:40.688722 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.688686    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"]
Jul 24 20:21:40.694227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.694191    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:40.697183 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.697164    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-wv5xh\""
Jul 24 20:21:40.698317 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.698294    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\""
Jul 24 20:21:40.699789 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.699767    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"]
Jul 24 20:21:40.703386 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.702898    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:40.705711 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.705683    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Jul 24 20:21:40.705819 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.705768    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-mmkb8\""
Jul 24 20:21:40.706708 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.706643    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"]
Jul 24 20:21:40.727779 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.727753    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"]
Jul 24 20:21:40.797790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.797744    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgxvw\" (UniqueName: \"kubernetes.io/projected/08120a05-023f-47c3-b1a7-04b8c880bc1e-kube-api-access-tgxvw\") pod \"llmisvc-controller-manager-777c6bd7d5-r9g9b\" (UID: \"08120a05-023f-47c3-b1a7-04b8c880bc1e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:40.797983 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.797824    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08120a05-023f-47c3-b1a7-04b8c880bc1e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r9g9b\" (UID: \"08120a05-023f-47c3-b1a7-04b8c880bc1e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:40.797983 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.797962    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:40.798092 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.797999    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvfrt\" (UniqueName: \"kubernetes.io/projected/b117a634-eb7f-49d8-84f9-0fccb697bbdf-kube-api-access-bvfrt\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:40.886062 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.886026    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:21:40.886142 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.886112    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:21:40.886183 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.886149    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:21:40.899286 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.899259    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:40.899374 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.899303    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvfrt\" (UniqueName: \"kubernetes.io/projected/b117a634-eb7f-49d8-84f9-0fccb697bbdf-kube-api-access-bvfrt\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:40.899374 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.899347    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgxvw\" (UniqueName: \"kubernetes.io/projected/08120a05-023f-47c3-b1a7-04b8c880bc1e-kube-api-access-tgxvw\") pod \"llmisvc-controller-manager-777c6bd7d5-r9g9b\" (UID: \"08120a05-023f-47c3-b1a7-04b8c880bc1e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:40.899482 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.899384    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08120a05-023f-47c3-b1a7-04b8c880bc1e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r9g9b\" (UID: \"08120a05-023f-47c3-b1a7-04b8c880bc1e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:40.899482 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:40.899420    2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 24 20:21:40.899619 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:40.899490    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert podName:b117a634-eb7f-49d8-84f9-0fccb697bbdf nodeName:}" failed. No retries permitted until 2026-07-24 20:21:41.399466773 +0000 UTC m=+376.325605660 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert") pod "kserve-controller-manager-d6f4b65b8-5sdvn" (UID: "b117a634-eb7f-49d8-84f9-0fccb697bbdf") : secret "kserve-webhook-server-cert" not found
Jul 24 20:21:40.899619 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:40.899553    2566 secret.go:189] Couldn't get secret opendatahub/llmisvc-webhook-server-cert: secret "llmisvc-webhook-server-cert" not found
Jul 24 20:21:40.899619 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:40.899606    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/08120a05-023f-47c3-b1a7-04b8c880bc1e-cert podName:08120a05-023f-47c3-b1a7-04b8c880bc1e nodeName:}" failed. No retries permitted until 2026-07-24 20:21:41.399590235 +0000 UTC m=+376.325729121 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/08120a05-023f-47c3-b1a7-04b8c880bc1e-cert") pod "llmisvc-controller-manager-777c6bd7d5-r9g9b" (UID: "08120a05-023f-47c3-b1a7-04b8c880bc1e") : secret "llmisvc-webhook-server-cert" not found
Jul 24 20:21:40.914425 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.914389    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvfrt\" (UniqueName: \"kubernetes.io/projected/b117a634-eb7f-49d8-84f9-0fccb697bbdf-kube-api-access-bvfrt\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:40.914425 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:40.914413    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgxvw\" (UniqueName: \"kubernetes.io/projected/08120a05-023f-47c3-b1a7-04b8c880bc1e-kube-api-access-tgxvw\") pod \"llmisvc-controller-manager-777c6bd7d5-r9g9b\" (UID: \"08120a05-023f-47c3-b1a7-04b8c880bc1e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:41.031274 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.031237    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c" event={"ID":"2c88b65a-ebb3-4dc9-a51c-3d813671b8df","Type":"ContainerStarted","Data":"d2eb68d4b03c5b1bb6e51fb4bd84512d2e5c29a4f99fb808e1eb8bdac664881b"}
Jul 24 20:21:41.087784 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.087716    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c" podStartSLOduration=1.600806807 podStartE2EDuration="4.087692168s" totalImagesPullingTime="2.486885361s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:41.086127741 +0000 UTC m=+376.012266663" watchObservedRunningTime="2026-07-24 20:21:41.087692168 +0000 UTC m=+376.013831074"
Jul 24 20:21:41.237080 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.236989    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:41.238660 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.238635    2566 patch_prober.go:28] interesting pod/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused" start-of-body=
Jul 24 20:21:41.238817 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.238698    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c" podUID="2c88b65a-ebb3-4dc9-a51c-3d813671b8df" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.19:15021/healthz/ready\": dial tcp 10.133.0.19:15021: connect: connection refused"
Jul 24 20:21:41.404366 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.404328    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08120a05-023f-47c3-b1a7-04b8c880bc1e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r9g9b\" (UID: \"08120a05-023f-47c3-b1a7-04b8c880bc1e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:41.404584 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.404402    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:41.404584 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:41.404495    2566 secret.go:189] Couldn't get secret opendatahub/kserve-webhook-server-cert: secret "kserve-webhook-server-cert" not found
Jul 24 20:21:41.404584 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:41.404568    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert podName:b117a634-eb7f-49d8-84f9-0fccb697bbdf nodeName:}" failed. No retries permitted until 2026-07-24 20:21:42.404554258 +0000 UTC m=+377.330693140 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "cert" (UniqueName: "kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert") pod "kserve-controller-manager-d6f4b65b8-5sdvn" (UID: "b117a634-eb7f-49d8-84f9-0fccb697bbdf") : secret "kserve-webhook-server-cert" not found
Jul 24 20:21:41.406884 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.406859    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/08120a05-023f-47c3-b1a7-04b8c880bc1e-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-r9g9b\" (UID: \"08120a05-023f-47c3-b1a7-04b8c880bc1e\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:41.618235 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.618197    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:41.754822 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:41.754798    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"]
Jul 24 20:21:41.756772 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:41.756745    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod08120a05_023f_47c3_b1a7_04b8c880bc1e.slice/crio-ec27b8fdeec0808c898fa056dc1fcc646ddd2f243a10a82fd68cd804e403e391 WatchSource:0}: Error finding container ec27b8fdeec0808c898fa056dc1fcc646ddd2f243a10a82fd68cd804e403e391: Status 404 returned error can't find the container with id ec27b8fdeec0808c898fa056dc1fcc646ddd2f243a10a82fd68cd804e403e391
Jul 24 20:21:42.036363 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:42.036278    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b" event={"ID":"08120a05-023f-47c3-b1a7-04b8c880bc1e","Type":"ContainerStarted","Data":"ec27b8fdeec0808c898fa056dc1fcc646ddd2f243a10a82fd68cd804e403e391"}
Jul 24 20:21:42.241049 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:42.241012    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:42.413128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:42.413090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:42.415690 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:42.415666    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/b117a634-eb7f-49d8-84f9-0fccb697bbdf-cert\") pod \"kserve-controller-manager-d6f4b65b8-5sdvn\" (UID: \"b117a634-eb7f-49d8-84f9-0fccb697bbdf\") " pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:42.509064 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:42.509032    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:42.655289 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:42.655182    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"]
Jul 24 20:21:42.829798 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:21:42.829730    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb117a634_eb7f_49d8_84f9_0fccb697bbdf.slice/crio-c47f21a25e43665ddb32fecd9858363a1eae7a23a67c527c99047d27a2feece0 WatchSource:0}: Error finding container c47f21a25e43665ddb32fecd9858363a1eae7a23a67c527c99047d27a2feece0: Status 404 returned error can't find the container with id c47f21a25e43665ddb32fecd9858363a1eae7a23a67c527c99047d27a2feece0
Jul 24 20:21:43.047322 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:43.047283    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn" event={"ID":"b117a634-eb7f-49d8-84f9-0fccb697bbdf","Type":"ContainerStarted","Data":"c47f21a25e43665ddb32fecd9858363a1eae7a23a67c527c99047d27a2feece0"}
Jul 24 20:21:43.047604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:43.047575    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:43.048698 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:43.048677    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c"
Jul 24 20:21:46.061343 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.061304    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b" event={"ID":"08120a05-023f-47c3-b1a7-04b8c880bc1e","Type":"ContainerStarted","Data":"060c8a4085a6a58929a4d9812bc3740ce5f887c67be770866a133b9b7c3aff15"}
Jul 24 20:21:46.061811 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.061433    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:21:46.062943 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.062925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn" event={"ID":"b117a634-eb7f-49d8-84f9-0fccb697bbdf","Type":"ContainerStarted","Data":"1969377eaf01acc646ec665ae7491c9f6d8e9be59e5b6a05649c908dd29a7f20"}
Jul 24 20:21:46.062995 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.062973    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:21:46.079434 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.079389    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b" podStartSLOduration=2.554069287 podStartE2EDuration="6.079377752s" totalImagesPullingTime="3.525308465s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:46.078136404 +0000 UTC m=+381.004275312" watchObservedRunningTime="2026-07-24 20:21:46.079377752 +0000 UTC m=+381.005516665"
Jul 24 20:21:46.094868 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.094824    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn" podStartSLOduration=3.642300412 podStartE2EDuration="6.094811302s" totalImagesPullingTime="2.45251089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:21:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:21:46.094101897 +0000 UTC m=+381.020240802" watchObservedRunningTime="2026-07-24 20:21:46.094811302 +0000 UTC m=+381.020950207"
Jul 24 20:21:46.730372 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.730338    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"]
Jul 24 20:21:46.730581 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.730555    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n" podUID="9b3b4886-da51-4d25-bd9f-c5a8846c9e93" containerName="discovery" containerID="cri-o://89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d" gracePeriod=30
Jul 24 20:21:46.967391 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:46.967367    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:47.054177 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054104    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-ca-configmap\") pod \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") "
Jul 24 20:21:47.054177 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054139    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-crgz8\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-kube-api-access-crgz8\") pod \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") "
Jul 24 20:21:47.054177 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054170    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-cacerts\") pod \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") "
Jul 24 20:21:47.054423 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054211    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-token\") pod \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") "
Jul 24 20:21:47.054423 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-dns-cert\") pod \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") "
Jul 24 20:21:47.054423 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054257    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-kubeconfig\") pod \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") "
Jul 24 20:21:47.054423 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054301    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-local-certs\") pod \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\" (UID: \"9b3b4886-da51-4d25-bd9f-c5a8846c9e93\") "
Jul 24 20:21:47.054632 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.054605    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "9b3b4886-da51-4d25-bd9f-c5a8846c9e93" (UID: "9b3b4886-da51-4d25-bd9f-c5a8846c9e93"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:21:47.056815 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.056773    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-token" (OuterVolumeSpecName: "istio-token") pod "9b3b4886-da51-4d25-bd9f-c5a8846c9e93" (UID: "9b3b4886-da51-4d25-bd9f-c5a8846c9e93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:21:47.056945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.056812    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "9b3b4886-da51-4d25-bd9f-c5a8846c9e93" (UID: "9b3b4886-da51-4d25-bd9f-c5a8846c9e93"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:21:47.056945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.056819    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-kube-api-access-crgz8" (OuterVolumeSpecName: "kube-api-access-crgz8") pod "9b3b4886-da51-4d25-bd9f-c5a8846c9e93" (UID: "9b3b4886-da51-4d25-bd9f-c5a8846c9e93"). InnerVolumeSpecName "kube-api-access-crgz8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:21:47.056945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.056861    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-local-certs" (OuterVolumeSpecName: "local-certs") pod "9b3b4886-da51-4d25-bd9f-c5a8846c9e93" (UID: "9b3b4886-da51-4d25-bd9f-c5a8846c9e93"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:21:47.056945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.056871    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-cacerts" (OuterVolumeSpecName: "cacerts") pod "9b3b4886-da51-4d25-bd9f-c5a8846c9e93" (UID: "9b3b4886-da51-4d25-bd9f-c5a8846c9e93"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:21:47.057172 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.057101    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "9b3b4886-da51-4d25-bd9f-c5a8846c9e93" (UID: "9b3b4886-da51-4d25-bd9f-c5a8846c9e93"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:21:47.068444 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.068419    2566 generic.go:356] "Generic (PLEG): container finished" podID="9b3b4886-da51-4d25-bd9f-c5a8846c9e93" containerID="89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d" exitCode=0
Jul 24 20:21:47.068843 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.068457    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"
Jul 24 20:21:47.068843 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.068502    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n" event={"ID":"9b3b4886-da51-4d25-bd9f-c5a8846c9e93","Type":"ContainerDied","Data":"89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d"}
Jul 24 20:21:47.068843 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.068575    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n" event={"ID":"9b3b4886-da51-4d25-bd9f-c5a8846c9e93","Type":"ContainerDied","Data":"fc14a8baf3690e393e585d1eb72740fd01b7a90d332a1b9cb74af6b20140cfef"}
Jul 24 20:21:47.068843 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.068604    2566 scope.go:117] "RemoveContainer" containerID="89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d"
Jul 24 20:21:47.076956 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.076938    2566 scope.go:117] "RemoveContainer" containerID="89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d"
Jul 24 20:21:47.077246 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:21:47.077223    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d\": container with ID starting with 89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d not found: ID does not exist" containerID="89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d"
Jul 24 20:21:47.077319 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.077256    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d"} err="failed to get container status \"89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d\": rpc error: code = NotFound desc = could not find container \"89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d\": container with ID starting with 89ade3a4b50e74b6a41040feed918c487be9798d73cdc30a2d511e4616319d5d not found: ID does not exist"
Jul 24 20:21:47.090343 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.090319    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"]
Jul 24 20:21:47.097072 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.097049    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-vtc6n"]
Jul 24 20:21:47.155334 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.155302    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-ca-configmap\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:21:47.155469 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.155439    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crgz8\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-kube-api-access-crgz8\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:21:47.155594 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.155574    2566 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-cacerts\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:21:47.155649 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.155598    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:21:47.155649 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.155610    2566 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-csr-dns-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:21:47.155649 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.155620    2566 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-istio-kubeconfig\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:21:47.155649 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.155632    2566 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/9b3b4886-da51-4d25-bd9f-c5a8846c9e93-local-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:21:47.692544 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:47.692505    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9b3b4886-da51-4d25-bd9f-c5a8846c9e93" path="/var/lib/kubelet/pods/9b3b4886-da51-4d25-bd9f-c5a8846c9e93/volumes"
Jul 24 20:21:48.019689 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:48.019605    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-7998c67dc4-wqdt7"
Jul 24 20:21:48.021691 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:21:48.021671    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-dtkrp"
Jul 24 20:22:17.071332 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:17.071297    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-r9g9b"
Jul 24 20:22:17.074300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:17.074280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-d6f4b65b8-5sdvn"
Jul 24 20:22:31.299755 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.299720    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5z9"]
Jul 24 20:22:31.300153 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.299978    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9b3b4886-da51-4d25-bd9f-c5a8846c9e93" containerName="discovery"
Jul 24 20:22:31.300153 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.299988    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9b3b4886-da51-4d25-bd9f-c5a8846c9e93" containerName="discovery"
Jul 24 20:22:31.300153 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.300059    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9b3b4886-da51-4d25-bd9f-c5a8846c9e93" containerName="discovery"
Jul 24 20:22:31.303056 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.303038    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9"
Jul 24 20:22:31.306797 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.306776    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Jul 24 20:22:31.308116 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.308095    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Jul 24 20:22:31.309074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.309051    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-catalog-dockercfg-x8k79\""
Jul 24 20:22:31.319136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.319112    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5z9"]
Jul 24 20:22:31.488126 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.488079    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gxmd7\" (UniqueName: \"kubernetes.io/projected/2acb0113-69bc-48d0-9f43-c3a54504b4e4-kube-api-access-gxmd7\") pod \"kuadrant-operator-catalog-fk5z9\" (UID: \"2acb0113-69bc-48d0-9f43-c3a54504b4e4\") " pod="kuadrant-system/kuadrant-operator-catalog-fk5z9"
Jul 24 20:22:31.589037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.588967    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gxmd7\" (UniqueName: \"kubernetes.io/projected/2acb0113-69bc-48d0-9f43-c3a54504b4e4-kube-api-access-gxmd7\") pod \"kuadrant-operator-catalog-fk5z9\" (UID: \"2acb0113-69bc-48d0-9f43-c3a54504b4e4\") " pod="kuadrant-system/kuadrant-operator-catalog-fk5z9"
Jul 24 20:22:31.596969 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.596944    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gxmd7\" (UniqueName: \"kubernetes.io/projected/2acb0113-69bc-48d0-9f43-c3a54504b4e4-kube-api-access-gxmd7\") pod \"kuadrant-operator-catalog-fk5z9\" (UID: \"2acb0113-69bc-48d0-9f43-c3a54504b4e4\") " pod="kuadrant-system/kuadrant-operator-catalog-fk5z9"
Jul 24 20:22:31.613456 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.613435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9"
Jul 24 20:22:31.658434 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.658407    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5z9"]
Jul 24 20:22:31.736098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.736039    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5z9"]
Jul 24 20:22:31.738194 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:22:31.738163    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2acb0113_69bc_48d0_9f43_c3a54504b4e4.slice/crio-c27db2b320493ee0d239bfb4a10fe61ee22b086c62c867fa0b028c3c92f6263d WatchSource:0}: Error finding container c27db2b320493ee0d239bfb4a10fe61ee22b086c62c867fa0b028c3c92f6263d: Status 404 returned error can't find the container with id c27db2b320493ee0d239bfb4a10fe61ee22b086c62c867fa0b028c3c92f6263d
Jul 24 20:22:31.873136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.873112    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6z6fb"]
Jul 24 20:22:31.876176 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.876158    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:31.885611 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.885339    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6z6fb"]
Jul 24 20:22:31.991843 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:31.991810    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c68cl\" (UniqueName: \"kubernetes.io/projected/2514a4a0-d6c1-44c3-8eb1-200208f0f627-kube-api-access-c68cl\") pod \"kuadrant-operator-catalog-6z6fb\" (UID: \"2514a4a0-d6c1-44c3-8eb1-200208f0f627\") " pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:32.092571 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:32.092544    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c68cl\" (UniqueName: \"kubernetes.io/projected/2514a4a0-d6c1-44c3-8eb1-200208f0f627-kube-api-access-c68cl\") pod \"kuadrant-operator-catalog-6z6fb\" (UID: \"2514a4a0-d6c1-44c3-8eb1-200208f0f627\") " pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:32.100665 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:32.100642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c68cl\" (UniqueName: \"kubernetes.io/projected/2514a4a0-d6c1-44c3-8eb1-200208f0f627-kube-api-access-c68cl\") pod \"kuadrant-operator-catalog-6z6fb\" (UID: \"2514a4a0-d6c1-44c3-8eb1-200208f0f627\") " pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:32.191617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:32.191559    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:32.246760 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:32.246693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9" event={"ID":"2acb0113-69bc-48d0-9f43-c3a54504b4e4","Type":"ContainerStarted","Data":"c27db2b320493ee0d239bfb4a10fe61ee22b086c62c867fa0b028c3c92f6263d"}
Jul 24 20:22:32.316509 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:32.316485    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-6z6fb"]
Jul 24 20:22:32.348752 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:22:32.348724    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2514a4a0_d6c1_44c3_8eb1_200208f0f627.slice/crio-63c4bcc9897a41cd1acb0ace241659eb2ff2a210871217efd3ea06dfd9aff9ba WatchSource:0}: Error finding container 63c4bcc9897a41cd1acb0ace241659eb2ff2a210871217efd3ea06dfd9aff9ba: Status 404 returned error can't find the container with id 63c4bcc9897a41cd1acb0ace241659eb2ff2a210871217efd3ea06dfd9aff9ba
Jul 24 20:22:33.253177 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:33.253137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb" event={"ID":"2514a4a0-d6c1-44c3-8eb1-200208f0f627","Type":"ContainerStarted","Data":"63c4bcc9897a41cd1acb0ace241659eb2ff2a210871217efd3ea06dfd9aff9ba"}
Jul 24 20:22:34.258435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.258398    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb" event={"ID":"2514a4a0-d6c1-44c3-8eb1-200208f0f627","Type":"ContainerStarted","Data":"3c9dac76ee5e397c7ee7f257f64fd901425b789f0be717163e89e3e7187dab1c"}
Jul 24 20:22:34.259824 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.259801    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9" event={"ID":"2acb0113-69bc-48d0-9f43-c3a54504b4e4","Type":"ContainerStarted","Data":"a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22"}
Jul 24 20:22:34.259915 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.259859    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9" podUID="2acb0113-69bc-48d0-9f43-c3a54504b4e4" containerName="registry-server" containerID="cri-o://a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22" gracePeriod=2
Jul 24 20:22:34.274432 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.274392    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb" podStartSLOduration=1.9435280480000001 podStartE2EDuration="3.274378382s" totalImagesPullingTime="1.330850334s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:22:34.272778541 +0000 UTC m=+429.198917447" watchObservedRunningTime="2026-07-24 20:22:34.274378382 +0000 UTC m=+429.200517286"
Jul 24 20:22:34.287539 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.287496    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9" podStartSLOduration=1.349125916 podStartE2EDuration="3.287483355s" totalImagesPullingTime="1.938357439s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:22:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:22:34.286175562 +0000 UTC m=+429.212314510" watchObservedRunningTime="2026-07-24 20:22:34.287483355 +0000 UTC m=+429.213622259"
Jul 24 20:22:34.508720 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.508661    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9"
Jul 24 20:22:34.512385 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.512359    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gxmd7\" (UniqueName: \"kubernetes.io/projected/2acb0113-69bc-48d0-9f43-c3a54504b4e4-kube-api-access-gxmd7\") pod \"2acb0113-69bc-48d0-9f43-c3a54504b4e4\" (UID: \"2acb0113-69bc-48d0-9f43-c3a54504b4e4\") "
Jul 24 20:22:34.514400 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.514379    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2acb0113-69bc-48d0-9f43-c3a54504b4e4-kube-api-access-gxmd7" (OuterVolumeSpecName: "kube-api-access-gxmd7") pod "2acb0113-69bc-48d0-9f43-c3a54504b4e4" (UID: "2acb0113-69bc-48d0-9f43-c3a54504b4e4"). InnerVolumeSpecName "kube-api-access-gxmd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:22:34.613652 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:34.613613    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gxmd7\" (UniqueName: \"kubernetes.io/projected/2acb0113-69bc-48d0-9f43-c3a54504b4e4-kube-api-access-gxmd7\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:22:35.265000 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.264962    2566 generic.go:356] "Generic (PLEG): container finished" podID="2acb0113-69bc-48d0-9f43-c3a54504b4e4" containerID="a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22" exitCode=0
Jul 24 20:22:35.265452 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.265007    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9"
Jul 24 20:22:35.265452 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.265047    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9" event={"ID":"2acb0113-69bc-48d0-9f43-c3a54504b4e4","Type":"ContainerDied","Data":"a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22"}
Jul 24 20:22:35.265452 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.265107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-fk5z9" event={"ID":"2acb0113-69bc-48d0-9f43-c3a54504b4e4","Type":"ContainerDied","Data":"c27db2b320493ee0d239bfb4a10fe61ee22b086c62c867fa0b028c3c92f6263d"}
Jul 24 20:22:35.265452 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.265136    2566 scope.go:117] "RemoveContainer" containerID="a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22"
Jul 24 20:22:35.273489 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.273475    2566 scope.go:117] "RemoveContainer" containerID="a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22"
Jul 24 20:22:35.273748 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:22:35.273730    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22\": container with ID starting with a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22 not found: ID does not exist" containerID="a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22"
Jul 24 20:22:35.273813 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.273761    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22"} err="failed to get container status \"a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22\": rpc error: code = NotFound desc = could not find container \"a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22\": container with ID starting with a457fde8edaec303a4466d99d357ab752558ac9aa94490bef0b409b23df29c22 not found: ID does not exist"
Jul 24 20:22:35.285094 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.285072    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5z9"]
Jul 24 20:22:35.288962 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.288934    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-fk5z9"]
Jul 24 20:22:35.691159 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:35.691116    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2acb0113-69bc-48d0-9f43-c3a54504b4e4" path="/var/lib/kubelet/pods/2acb0113-69bc-48d0-9f43-c3a54504b4e4/volumes"
Jul 24 20:22:42.192259 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:42.192214    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:42.192259 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:42.192280    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:42.213280 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:42.213250    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:22:42.313079 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:22:42.313047    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-6z6fb"
Jul 24 20:23:06.065731 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.065696    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"]
Jul 24 20:23:06.066150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.065975    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2acb0113-69bc-48d0-9f43-c3a54504b4e4" containerName="registry-server"
Jul 24 20:23:06.066150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.065985    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2acb0113-69bc-48d0-9f43-c3a54504b4e4" containerName="registry-server"
Jul 24 20:23:06.066150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.066065    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2acb0113-69bc-48d0-9f43-c3a54504b4e4" containerName="registry-server"
Jul 24 20:23:06.068883 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.068868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"
Jul 24 20:23:06.071854 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.071830    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-operator-dockercfg-6rwjh\""
Jul 24 20:23:06.082815 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.082786    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"]
Jul 24 20:23:06.161192 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.161153    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkqtv\" (UniqueName: \"kubernetes.io/projected/9fddb5b2-97a3-470a-9d0a-7d562ce39cac-kube-api-access-wkqtv\") pod \"authorino-operator-75cc8d5cb7-88qk8\" (UID: \"9fddb5b2-97a3-470a-9d0a-7d562ce39cac\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"
Jul 24 20:23:06.261771 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.261738    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkqtv\" (UniqueName: \"kubernetes.io/projected/9fddb5b2-97a3-470a-9d0a-7d562ce39cac-kube-api-access-wkqtv\") pod \"authorino-operator-75cc8d5cb7-88qk8\" (UID: \"9fddb5b2-97a3-470a-9d0a-7d562ce39cac\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"
Jul 24 20:23:06.275187 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.275152    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkqtv\" (UniqueName: \"kubernetes.io/projected/9fddb5b2-97a3-470a-9d0a-7d562ce39cac-kube-api-access-wkqtv\") pod \"authorino-operator-75cc8d5cb7-88qk8\" (UID: \"9fddb5b2-97a3-470a-9d0a-7d562ce39cac\") " pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"
Jul 24 20:23:06.379312 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.379280    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"
Jul 24 20:23:06.514905 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:06.514876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"]
Jul 24 20:23:06.517060 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:06.517024    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9fddb5b2_97a3_470a_9d0a_7d562ce39cac.slice/crio-c7ea4da7afd19883817833f828ee2aabf98ae37b4432695da818d902413e9842 WatchSource:0}: Error finding container c7ea4da7afd19883817833f828ee2aabf98ae37b4432695da818d902413e9842: Status 404 returned error can't find the container with id c7ea4da7afd19883817833f828ee2aabf98ae37b4432695da818d902413e9842
Jul 24 20:23:07.393648 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:07.393590    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8" event={"ID":"9fddb5b2-97a3-470a-9d0a-7d562ce39cac","Type":"ContainerStarted","Data":"c7ea4da7afd19883817833f828ee2aabf98ae37b4432695da818d902413e9842"}
Jul 24 20:23:09.266833 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.266799    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"]
Jul 24 20:23:09.270087 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.270066    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"
Jul 24 20:23:09.272870 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.272847    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-xsrrd\""
Jul 24 20:23:09.272975 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.272851    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Jul 24 20:23:09.286705 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.286678    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"]
Jul 24 20:23:09.387192 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.387158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-445gt\" (UniqueName: \"kubernetes.io/projected/a485a4a5-3a3b-45f4-8949-ef5bdcaeb992-kube-api-access-445gt\") pod \"dns-operator-controller-manager-864dd78c4c-28x78\" (UID: \"a485a4a5-3a3b-45f4-8949-ef5bdcaeb992\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"
Jul 24 20:23:09.405694 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.405615    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8" event={"ID":"9fddb5b2-97a3-470a-9d0a-7d562ce39cac","Type":"ContainerStarted","Data":"22fc3e1c2b9b0b8391d43b0c160eb711d83c332f0065785ab7ab830601dc77aa"}
Jul 24 20:23:09.405876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.405761    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"
Jul 24 20:23:09.427387 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.427332    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8" podStartSLOduration=1.19854329 podStartE2EDuration="3.427318439s" totalImagesPullingTime="2.228775149s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:23:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:23:09.426213104 +0000 UTC m=+464.352352012" watchObservedRunningTime="2026-07-24 20:23:09.427318439 +0000 UTC m=+464.353457344"
Jul 24 20:23:09.488067 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.488021    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-445gt\" (UniqueName: \"kubernetes.io/projected/a485a4a5-3a3b-45f4-8949-ef5bdcaeb992-kube-api-access-445gt\") pod \"dns-operator-controller-manager-864dd78c4c-28x78\" (UID: \"a485a4a5-3a3b-45f4-8949-ef5bdcaeb992\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"
Jul 24 20:23:09.505941 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.505906    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-445gt\" (UniqueName: \"kubernetes.io/projected/a485a4a5-3a3b-45f4-8949-ef5bdcaeb992-kube-api-access-445gt\") pod \"dns-operator-controller-manager-864dd78c4c-28x78\" (UID: \"a485a4a5-3a3b-45f4-8949-ef5bdcaeb992\") " pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"
Jul 24 20:23:09.579815 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.579715    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"
Jul 24 20:23:09.739027 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:09.738987    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"]
Jul 24 20:23:09.741848 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:09.741821    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda485a4a5_3a3b_45f4_8949_ef5bdcaeb992.slice/crio-162f378aa1060ed20aeb1dfe2a07412fe68d0d29814f631cc577e165b2119ba8 WatchSource:0}: Error finding container 162f378aa1060ed20aeb1dfe2a07412fe68d0d29814f631cc577e165b2119ba8: Status 404 returned error can't find the container with id 162f378aa1060ed20aeb1dfe2a07412fe68d0d29814f631cc577e165b2119ba8
Jul 24 20:23:10.411393 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:10.411355    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78" event={"ID":"a485a4a5-3a3b-45f4-8949-ef5bdcaeb992","Type":"ContainerStarted","Data":"162f378aa1060ed20aeb1dfe2a07412fe68d0d29814f631cc577e165b2119ba8"}
Jul 24 20:23:12.422445 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.422408    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78" event={"ID":"a485a4a5-3a3b-45f4-8949-ef5bdcaeb992","Type":"ContainerStarted","Data":"1b414679d4688202d612e1aabd8586a7bc087dd3accbea4c184fbc04a5d400b8"}
Jul 24 20:23:12.422872 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.422480    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"
Jul 24 20:23:12.438270 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.438220    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78" podStartSLOduration=1.516543316 podStartE2EDuration="3.438208162s" totalImagesPullingTime="1.921664846s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:23:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:23:12.436858935 +0000 UTC m=+467.362997840" watchObservedRunningTime="2026-07-24 20:23:12.438208162 +0000 UTC m=+467.364347064"
Jul 24 20:23:12.853637 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.853604    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"]
Jul 24 20:23:12.856888 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.856866    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:12.867631 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.867611    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-kzxq2\""
Jul 24 20:23:12.874815 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.874791    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"]
Jul 24 20:23:12.917561 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:12.917496    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wpvrz\" (UniqueName: \"kubernetes.io/projected/34914530-afb4-4294-aa0c-a65cd4fefd31-kube-api-access-wpvrz\") pod \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" (UID: \"34914530-afb4-4294-aa0c-a65cd4fefd31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:13.018661 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:13.018620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wpvrz\" (UniqueName: \"kubernetes.io/projected/34914530-afb4-4294-aa0c-a65cd4fefd31-kube-api-access-wpvrz\") pod \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" (UID: \"34914530-afb4-4294-aa0c-a65cd4fefd31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:13.036116 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:13.036086    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wpvrz\" (UniqueName: \"kubernetes.io/projected/34914530-afb4-4294-aa0c-a65cd4fefd31-kube-api-access-wpvrz\") pod \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" (UID: \"34914530-afb4-4294-aa0c-a65cd4fefd31\") " pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:13.167270 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:13.167180    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:13.295163 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:13.295091    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"]
Jul 24 20:23:13.297313 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:13.297281    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34914530_afb4_4294_aa0c_a65cd4fefd31.slice/crio-f184c1d0fcc07bf463e3cd6fb32bc6d0b5a5ba9baa28e82473c2b472b7a79d60 WatchSource:0}: Error finding container f184c1d0fcc07bf463e3cd6fb32bc6d0b5a5ba9baa28e82473c2b472b7a79d60: Status 404 returned error can't find the container with id f184c1d0fcc07bf463e3cd6fb32bc6d0b5a5ba9baa28e82473c2b472b7a79d60
Jul 24 20:23:13.430735 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:13.430649    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl" event={"ID":"34914530-afb4-4294-aa0c-a65cd4fefd31","Type":"ContainerStarted","Data":"f184c1d0fcc07bf463e3cd6fb32bc6d0b5a5ba9baa28e82473c2b472b7a79d60"}
Jul 24 20:23:15.441469 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:15.441433    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl" event={"ID":"34914530-afb4-4294-aa0c-a65cd4fefd31","Type":"ContainerStarted","Data":"59d68ccb230935593bfd96ebe80bc4cd16692eeb8351c8881a7859b81a95f564"}
Jul 24 20:23:15.441875 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:15.441502    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:15.459611 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:15.459516    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl" podStartSLOduration=1.7293718120000001 podStartE2EDuration="3.459500283s" totalImagesPullingTime="1.730128471s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:23:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:23:15.459294235 +0000 UTC m=+470.385433161" watchObservedRunningTime="2026-07-24 20:23:15.459500283 +0000 UTC m=+470.385639190"
Jul 24 20:23:17.710400 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.710367    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"]
Jul 24 20:23:17.713504 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.713486    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:17.716172 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.716155    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wnd75\""
Jul 24 20:23:17.728106 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.728084    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"]
Jul 24 20:23:17.862161 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.862125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvdmx\" (UniqueName: \"kubernetes.io/projected/543e880e-a85f-4c05-baa7-50a3ccb9c95a-kube-api-access-vvdmx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:17.862161 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.862170    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/543e880e-a85f-4c05-baa7-50a3ccb9c95a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:17.962727 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.962620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/543e880e-a85f-4c05-baa7-50a3ccb9c95a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:17.962907 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.962757    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvdmx\" (UniqueName: \"kubernetes.io/projected/543e880e-a85f-4c05-baa7-50a3ccb9c95a-kube-api-access-vvdmx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:17.963014 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.962993    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/543e880e-a85f-4c05-baa7-50a3ccb9c95a-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:17.975542 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:17.975496    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvdmx\" (UniqueName: \"kubernetes.io/projected/543e880e-a85f-4c05-baa7-50a3ccb9c95a-kube-api-access-vvdmx\") pod \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:18.023074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:18.023047    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:18.168470 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:18.168434    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"]
Jul 24 20:23:18.170757 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:18.170733    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod543e880e_a85f_4c05_baa7_50a3ccb9c95a.slice/crio-3f1e56499e2dd366f54ee1a6bbd292c593f50393ab5454a55363b68121ec79d9 WatchSource:0}: Error finding container 3f1e56499e2dd366f54ee1a6bbd292c593f50393ab5454a55363b68121ec79d9: Status 404 returned error can't find the container with id 3f1e56499e2dd366f54ee1a6bbd292c593f50393ab5454a55363b68121ec79d9
Jul 24 20:23:18.454848 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:18.454809    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7" event={"ID":"543e880e-a85f-4c05-baa7-50a3ccb9c95a","Type":"ContainerStarted","Data":"3f1e56499e2dd366f54ee1a6bbd292c593f50393ab5454a55363b68121ec79d9"}
Jul 24 20:23:20.414233 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:20.414202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/authorino-operator-75cc8d5cb7-88qk8"
Jul 24 20:23:23.433092 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:23.433062    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-864dd78c4c-28x78"
Jul 24 20:23:24.488257 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:24.488172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7" event={"ID":"543e880e-a85f-4c05-baa7-50a3ccb9c95a","Type":"ContainerStarted","Data":"fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185"}
Jul 24 20:23:24.488617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:24.488292    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:24.509662 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:24.509612    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7" podStartSLOduration=1.5390466649999999 podStartE2EDuration="7.509595053s" totalImagesPullingTime="5.970548388s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:23:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:23:24.507900464 +0000 UTC m=+479.434039405" watchObservedRunningTime="2026-07-24 20:23:24.509595053 +0000 UTC m=+479.435733962"
Jul 24 20:23:26.448573 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:26.448516    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:35.494839 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:35.494809    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:36.418838 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.418806    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"]
Jul 24 20:23:36.423219 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.423197    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.438050 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.438023    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"]
Jul 24 20:23:36.520695 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.520661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11c58b6-e105-4d61-a266-86acaae7740e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.521085 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.520723    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lj5tb\" (UniqueName: \"kubernetes.io/projected/e11c58b6-e105-4d61-a266-86acaae7740e-kube-api-access-lj5tb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.621135 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.621103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11c58b6-e105-4d61-a266-86acaae7740e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.621289 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.621177    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lj5tb\" (UniqueName: \"kubernetes.io/projected/e11c58b6-e105-4d61-a266-86acaae7740e-kube-api-access-lj5tb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.621577 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.621557    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11c58b6-e105-4d61-a266-86acaae7740e-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.634045 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.634022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lj5tb\" (UniqueName: \"kubernetes.io/projected/e11c58b6-e105-4d61-a266-86acaae7740e-kube-api-access-lj5tb\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.733429 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.733346    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:36.867897 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:36.867876    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"]
Jul 24 20:23:36.870490 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:36.870461    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode11c58b6_e105_4d61_a266_86acaae7740e.slice/crio-8b7524e707f4436563f38f47793aa9ac5e7be2959c00c9555263be34ce6ac42a WatchSource:0}: Error finding container 8b7524e707f4436563f38f47793aa9ac5e7be2959c00c9555263be34ce6ac42a: Status 404 returned error can't find the container with id 8b7524e707f4436563f38f47793aa9ac5e7be2959c00c9555263be34ce6ac42a
Jul 24 20:23:37.252339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.252298    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"]
Jul 24 20:23:37.258792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.258763    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"]
Jul 24 20:23:37.271363 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.271340    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"]
Jul 24 20:23:37.271573 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.271550    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" containerName="manager" containerID="cri-o://fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185" gracePeriod=2
Jul 24 20:23:37.276402 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.276379    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"]
Jul 24 20:23:37.276738 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.276717    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" containerName="manager"
Jul 24 20:23:37.276738 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.276734    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" containerName="manager"
Jul 24 20:23:37.276910 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.276805    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" containerName="manager"
Jul 24 20:23:37.280274 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.280255    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.286468 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.286441    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"]
Jul 24 20:23:37.293633 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.293596    2566 status_manager.go:1041] "Failed to update status for pod" err="failed to patch status \"{\\\"metadata\\\":{\\\"uid\\\":\\\"543e880e-a85f-4c05-baa7-50a3ccb9c95a\\\"},\\\"status\\\":{\\\"$setElementOrder/conditions\\\":[{\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"type\\\":\\\"Initialized\\\"},{\\\"type\\\":\\\"Ready\\\"},{\\\"type\\\":\\\"ContainersReady\\\"},{\\\"type\\\":\\\"PodScheduled\\\"}],\\\"conditions\\\":[{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodReadyToStartContainers\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Initialized\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"Ready\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"ContainersReady\\\"},{\\\"observedGeneration\\\":2,\\\"type\\\":\\\"PodScheduled\\\"}],\\\"observedGeneration\\\":2}}\" for pod \"kuadrant-system\"/\"kuadrant-operator-controller-manager-8684c9ff49-6llg7\": pods \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" not found" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.293888 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.293869    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"]
Jul 24 20:23:37.295932 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.295797    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.299571 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.299550    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"]
Jul 24 20:23:37.299987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.299970    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" containerName="manager"
Jul 24 20:23:37.300034 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.299991    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" containerName="manager"
Jul 24 20:23:37.300099 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.300088    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" containerName="manager"
Jul 24 20:23:37.303000 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.302983    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"]
Jul 24 20:23:37.303099 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.303083    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.303152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.303136    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" containerName="manager" containerID="cri-o://59d68ccb230935593bfd96ebe80bc4cd16692eeb8351c8881a7859b81a95f564" gracePeriod=2
Jul 24 20:23:37.326931 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.326859    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"]
Jul 24 20:23:37.327738 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.327699    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"]
Jul 24 20:23:37.345841 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.345809    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:37.347842 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.347805    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.428090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.428059    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588e9d0d-0551-4fdc-8f1e-4892dff72a04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vsmzd\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.428213 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.428104    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-spkqv\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.428213 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.428158    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vtvmw\" (UniqueName: \"kubernetes.io/projected/588e9d0d-0551-4fdc-8f1e-4892dff72a04-kube-api-access-vtvmw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vsmzd\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.428213 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.428179    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wtjnq\" (UniqueName: \"kubernetes.io/projected/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-kube-api-access-wtjnq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-spkqv\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.529890    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588e9d0d-0551-4fdc-8f1e-4892dff72a04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vsmzd\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.529947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-spkqv\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.530013    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vtvmw\" (UniqueName: \"kubernetes.io/projected/588e9d0d-0551-4fdc-8f1e-4892dff72a04-kube-api-access-vtvmw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vsmzd\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.530040    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wtjnq\" (UniqueName: \"kubernetes.io/projected/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-kube-api-access-wtjnq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-spkqv\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.530897    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-spkqv\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.531418    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588e9d0d-0551-4fdc-8f1e-4892dff72a04-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vsmzd\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.531842    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.533920    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.536667 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.535836    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:37.548604 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.548581    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wtjnq\" (UniqueName: \"kubernetes.io/projected/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-kube-api-access-wtjnq\") pod \"kuadrant-operator-controller-manager-6fb7879bc9-spkqv\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") " pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.548817 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.548796    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vtvmw\" (UniqueName: \"kubernetes.io/projected/588e9d0d-0551-4fdc-8f1e-4892dff72a04-kube-api-access-vtvmw\") pod \"kuadrant-operator-controller-manager-8684c9ff49-vsmzd\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") " pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.551651 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.551624    2566 generic.go:356] "Generic (PLEG): container finished" podID="34914530-afb4-4294-aa0c-a65cd4fefd31" containerID="59d68ccb230935593bfd96ebe80bc4cd16692eeb8351c8881a7859b81a95f564" exitCode=0
Jul 24 20:23:37.553201 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.553181    2566 generic.go:356] "Generic (PLEG): container finished" podID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" containerID="fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185" exitCode=0
Jul 24 20:23:37.553311 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.553234    2566 scope.go:117] "RemoveContainer" containerID="fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185"
Jul 24 20:23:37.553311 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.553245    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.555381 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.555362    2566 kuberuntime_manager.go:649] "Missing actuated resource record" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk" container="manager"
Jul 24 20:23:37.556191 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.556168    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:37.558407 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.558380    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.560436 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.560412    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:37.562392 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.562365    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:37.563238 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.563222    2566 scope.go:117] "RemoveContainer" containerID="fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185"
Jul 24 20:23:37.563481 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.563465    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185\": container with ID starting with fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185 not found: ID does not exist" containerID="fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185"
Jul 24 20:23:37.563562 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.563487    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185"} err="failed to get container status \"fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185\": rpc error: code = NotFound desc = could not find container \"fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185\": container with ID starting with fab8f4e834636d72834ec4eff32882d471ab6335fe4804cf7561bbfaf542c185 not found: ID does not exist"
Jul 24 20:23:37.564300 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.564284    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.565348 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.565335    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:37.567320 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.567299    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:37.569167 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.569144    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-859d7b78bd-lkgsl\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:37.571091 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:37.571071    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-8684c9ff49-6llg7\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-6llg7"
Jul 24 20:23:37.630308 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.630282    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/543e880e-a85f-4c05-baa7-50a3ccb9c95a-extensions-socket-volume\") pod \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") "
Jul 24 20:23:37.630417 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.630378    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvdmx\" (UniqueName: \"kubernetes.io/projected/543e880e-a85f-4c05-baa7-50a3ccb9c95a-kube-api-access-vvdmx\") pod \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\" (UID: \"543e880e-a85f-4c05-baa7-50a3ccb9c95a\") "
Jul 24 20:23:37.630656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.630629    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/543e880e-a85f-4c05-baa7-50a3ccb9c95a-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "543e880e-a85f-4c05-baa7-50a3ccb9c95a" (UID: "543e880e-a85f-4c05-baa7-50a3ccb9c95a"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:23:37.632179 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.632158    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/543e880e-a85f-4c05-baa7-50a3ccb9c95a-kube-api-access-vvdmx" (OuterVolumeSpecName: "kube-api-access-vvdmx") pod "543e880e-a85f-4c05-baa7-50a3ccb9c95a" (UID: "543e880e-a85f-4c05-baa7-50a3ccb9c95a"). InnerVolumeSpecName "kube-api-access-vvdmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:23:37.670461 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.670437    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:37.687221 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.687200    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:37.693391 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.693364    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="543e880e-a85f-4c05-baa7-50a3ccb9c95a" path="/var/lib/kubelet/pods/543e880e-a85f-4c05-baa7-50a3ccb9c95a/volumes"
Jul 24 20:23:37.732030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.731687    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wpvrz\" (UniqueName: \"kubernetes.io/projected/34914530-afb4-4294-aa0c-a65cd4fefd31-kube-api-access-wpvrz\") pod \"34914530-afb4-4294-aa0c-a65cd4fefd31\" (UID: \"34914530-afb4-4294-aa0c-a65cd4fefd31\") "
Jul 24 20:23:37.732030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.731985    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/543e880e-a85f-4c05-baa7-50a3ccb9c95a-extensions-socket-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:23:37.732030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.732006    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvdmx\" (UniqueName: \"kubernetes.io/projected/543e880e-a85f-4c05-baa7-50a3ccb9c95a-kube-api-access-vvdmx\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:23:37.734317 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.734269    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34914530-afb4-4294-aa0c-a65cd4fefd31-kube-api-access-wpvrz" (OuterVolumeSpecName: "kube-api-access-wpvrz") pod "34914530-afb4-4294-aa0c-a65cd4fefd31" (UID: "34914530-afb4-4294-aa0c-a65cd4fefd31"). InnerVolumeSpecName "kube-api-access-wpvrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:23:37.821656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.821631    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"]
Jul 24 20:23:37.823578 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:37.823551    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6134c8b6_3cf3_447b_8a90_cedfa0d7a9b1.slice/crio-7cebc82efc4d846fd60f08db774672fa7afb8efbdd4758939522a670c630825f WatchSource:0}: Error finding container 7cebc82efc4d846fd60f08db774672fa7afb8efbdd4758939522a670c630825f: Status 404 returned error can't find the container with id 7cebc82efc4d846fd60f08db774672fa7afb8efbdd4758939522a670c630825f
Jul 24 20:23:37.833145 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.833117    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wpvrz\" (UniqueName: \"kubernetes.io/projected/34914530-afb4-4294-aa0c-a65cd4fefd31-kube-api-access-wpvrz\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:23:37.846495 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:37.846475    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"]
Jul 24 20:23:37.852477 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:37.852454    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod588e9d0d_0551_4fdc_8f1e_4892dff72a04.slice/crio-330743048f983482db403ecbffb3efa2080975b9e893ded37b024f0266fd8f4c WatchSource:0}: Error finding container 330743048f983482db403ecbffb3efa2080975b9e893ded37b024f0266fd8f4c: Status 404 returned error can't find the container with id 330743048f983482db403ecbffb3efa2080975b9e893ded37b024f0266fd8f4c
Jul 24 20:23:38.561187 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.561153    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd" event={"ID":"588e9d0d-0551-4fdc-8f1e-4892dff72a04","Type":"ContainerStarted","Data":"41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524"}
Jul 24 20:23:38.561644 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.561214    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:38.561644 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.561239    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd" event={"ID":"588e9d0d-0551-4fdc-8f1e-4892dff72a04","Type":"ContainerStarted","Data":"330743048f983482db403ecbffb3efa2080975b9e893ded37b024f0266fd8f4c"}
Jul 24 20:23:38.562519 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.562497    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-859d7b78bd-lkgsl"
Jul 24 20:23:38.562519 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.562517    2566 scope.go:117] "RemoveContainer" containerID="59d68ccb230935593bfd96ebe80bc4cd16692eeb8351c8881a7859b81a95f564"
Jul 24 20:23:38.565101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.565078    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv" event={"ID":"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1","Type":"ContainerStarted","Data":"6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0"}
Jul 24 20:23:38.565228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.565120    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv" event={"ID":"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1","Type":"ContainerStarted","Data":"7cebc82efc4d846fd60f08db774672fa7afb8efbdd4758939522a670c630825f"}
Jul 24 20:23:38.565228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.565150    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:38.566855 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.566824    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" containerName="manager" containerID="cri-o://1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1" gracePeriod=2
Jul 24 20:23:38.651327 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.651269    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd" podStartSLOduration=1.651252927 podStartE2EDuration="1.651252927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:23:38.596722228 +0000 UTC m=+493.522861146" watchObservedRunningTime="2026-07-24 20:23:38.651252927 +0000 UTC m=+493.577391835"
Jul 24 20:23:38.651435 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.651405    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv" podStartSLOduration=1.651399231 podStartE2EDuration="1.651399231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:23:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:23:38.64483589 +0000 UTC m=+493.570974799" watchObservedRunningTime="2026-07-24 20:23:38.651399231 +0000 UTC m=+493.577538203"
Jul 24 20:23:38.815394 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.815336    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:38.817712 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:38.817686    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:38.942973 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.942945    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11c58b6-e105-4d61-a266-86acaae7740e-extensions-socket-volume\") pod \"e11c58b6-e105-4d61-a266-86acaae7740e\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") "
Jul 24 20:23:38.943133 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.943011    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lj5tb\" (UniqueName: \"kubernetes.io/projected/e11c58b6-e105-4d61-a266-86acaae7740e-kube-api-access-lj5tb\") pod \"e11c58b6-e105-4d61-a266-86acaae7740e\" (UID: \"e11c58b6-e105-4d61-a266-86acaae7740e\") "
Jul 24 20:23:38.943381 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.943351    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e11c58b6-e105-4d61-a266-86acaae7740e-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "e11c58b6-e105-4d61-a266-86acaae7740e" (UID: "e11c58b6-e105-4d61-a266-86acaae7740e"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:23:38.944999 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:38.944970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e11c58b6-e105-4d61-a266-86acaae7740e-kube-api-access-lj5tb" (OuterVolumeSpecName: "kube-api-access-lj5tb") pod "e11c58b6-e105-4d61-a266-86acaae7740e" (UID: "e11c58b6-e105-4d61-a266-86acaae7740e"). InnerVolumeSpecName "kube-api-access-lj5tb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:23:39.043633 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.043604    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lj5tb\" (UniqueName: \"kubernetes.io/projected/e11c58b6-e105-4d61-a266-86acaae7740e-kube-api-access-lj5tb\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:23:39.043633 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.043626    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/e11c58b6-e105-4d61-a266-86acaae7740e-extensions-socket-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:23:39.577493 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.577452    2566 generic.go:356] "Generic (PLEG): container finished" podID="e11c58b6-e105-4d61-a266-86acaae7740e" containerID="1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1" exitCode=0
Jul 24 20:23:39.577493 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.577495    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:39.577992 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.577505    2566 scope.go:117] "RemoveContainer" containerID="1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1"
Jul 24 20:23:39.580202 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:39.580177    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:39.586521 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.586506    2566 scope.go:117] "RemoveContainer" containerID="1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1"
Jul 24 20:23:39.586815 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:39.586799    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1\": container with ID starting with 1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1 not found: ID does not exist" containerID="1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1"
Jul 24 20:23:39.586881 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.586822    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1"} err="failed to get container status \"1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1\": rpc error: code = NotFound desc = could not find container \"1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1\": container with ID starting with 1fb4cf60bad49c9508f64a17ba215777ec44c840c967dba23b37b721a730b7e1 not found: ID does not exist"
Jul 24 20:23:39.588024 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:39.588005    2566 status_manager.go:1018] "Failed to get status for pod" err="pods \"kuadrant-operator-controller-manager-6fb7879bc9-rf8dk\" is forbidden: User \"system:node:ip-10-0-129-245.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-129-245.ec2.internal' and this object" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-rf8dk"
Jul 24 20:23:39.692274 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.692246    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" path="/var/lib/kubelet/pods/34914530-afb4-4294-aa0c-a65cd4fefd31/volumes"
Jul 24 20:23:39.692664 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:39.692646    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e11c58b6-e105-4d61-a266-86acaae7740e" path="/var/lib/kubelet/pods/e11c58b6-e105-4d61-a266-86acaae7740e/volumes"
Jul 24 20:23:49.580955 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.580923    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:23:49.581394 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.580999    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:49.658142 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.658106    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"]
Jul 24 20:23:49.658332 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.658307    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd" podUID="588e9d0d-0551-4fdc-8f1e-4892dff72a04" containerName="manager" containerID="cri-o://41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524" gracePeriod=10
Jul 24 20:23:49.874732 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.874695    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"]
Jul 24 20:23:49.875178 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.875163    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" containerName="manager"
Jul 24 20:23:49.875220 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.875182    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" containerName="manager"
Jul 24 20:23:49.875280 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.875269    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="34914530-afb4-4294-aa0c-a65cd4fefd31" containerName="manager"
Jul 24 20:23:49.879421 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.879399    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:49.891913 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.891887    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"]
Jul 24 20:23:49.912192 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.912168    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:49.922007 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.921978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvz5\" (UniqueName: \"kubernetes.io/projected/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-kube-api-access-jpvz5\") pod \"kuadrant-operator-controller-manager-86878c769f-tcprb\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:49.922120 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:49.922015    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tcprb\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:50.023077 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.023046    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588e9d0d-0551-4fdc-8f1e-4892dff72a04-extensions-socket-volume\") pod \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") "
Jul 24 20:23:50.023234 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.023112    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vtvmw\" (UniqueName: \"kubernetes.io/projected/588e9d0d-0551-4fdc-8f1e-4892dff72a04-kube-api-access-vtvmw\") pod \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\" (UID: \"588e9d0d-0551-4fdc-8f1e-4892dff72a04\") "
Jul 24 20:23:50.023272 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.023235    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvz5\" (UniqueName: \"kubernetes.io/projected/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-kube-api-access-jpvz5\") pod \"kuadrant-operator-controller-manager-86878c769f-tcprb\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:50.023309 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.023284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tcprb\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:50.023479 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.023454    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/588e9d0d-0551-4fdc-8f1e-4892dff72a04-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "588e9d0d-0551-4fdc-8f1e-4892dff72a04" (UID: "588e9d0d-0551-4fdc-8f1e-4892dff72a04"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:23:50.023718 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.023700    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-tcprb\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:50.025167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.025143    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/588e9d0d-0551-4fdc-8f1e-4892dff72a04-kube-api-access-vtvmw" (OuterVolumeSpecName: "kube-api-access-vtvmw") pod "588e9d0d-0551-4fdc-8f1e-4892dff72a04" (UID: "588e9d0d-0551-4fdc-8f1e-4892dff72a04"). InnerVolumeSpecName "kube-api-access-vtvmw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:23:50.031895 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.031873    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvz5\" (UniqueName: \"kubernetes.io/projected/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-kube-api-access-jpvz5\") pod \"kuadrant-operator-controller-manager-86878c769f-tcprb\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:50.124568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.124511    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/588e9d0d-0551-4fdc-8f1e-4892dff72a04-extensions-socket-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:23:50.124568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.124559    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vtvmw\" (UniqueName: \"kubernetes.io/projected/588e9d0d-0551-4fdc-8f1e-4892dff72a04-kube-api-access-vtvmw\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:23:50.190631 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.190512    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:50.321670 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.318796    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"]
Jul 24 20:23:50.326272 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:23:50.326241    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1915e104_0bb9_43cb_a7fa_b4b0a82586ce.slice/crio-a730a48f699db787ef4eaaa8243ca5611b2d8df43cc74d061e0d9787f1d769a0 WatchSource:0}: Error finding container a730a48f699db787ef4eaaa8243ca5611b2d8df43cc74d061e0d9787f1d769a0: Status 404 returned error can't find the container with id a730a48f699db787ef4eaaa8243ca5611b2d8df43cc74d061e0d9787f1d769a0
Jul 24 20:23:50.627148 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.627115    2566 generic.go:356] "Generic (PLEG): container finished" podID="588e9d0d-0551-4fdc-8f1e-4892dff72a04" containerID="41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524" exitCode=0
Jul 24 20:23:50.627634 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.627175    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"
Jul 24 20:23:50.627634 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.627190    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd" event={"ID":"588e9d0d-0551-4fdc-8f1e-4892dff72a04","Type":"ContainerDied","Data":"41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524"}
Jul 24 20:23:50.627634 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.627244    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd" event={"ID":"588e9d0d-0551-4fdc-8f1e-4892dff72a04","Type":"ContainerDied","Data":"330743048f983482db403ecbffb3efa2080975b9e893ded37b024f0266fd8f4c"}
Jul 24 20:23:50.627634 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.627269    2566 scope.go:117] "RemoveContainer" containerID="41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524"
Jul 24 20:23:50.628991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.628955    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb" event={"ID":"1915e104-0bb9-43cb-a7fa-b4b0a82586ce","Type":"ContainerStarted","Data":"b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f"}
Jul 24 20:23:50.629107 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.629011    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb" event={"ID":"1915e104-0bb9-43cb-a7fa-b4b0a82586ce","Type":"ContainerStarted","Data":"a730a48f699db787ef4eaaa8243ca5611b2d8df43cc74d061e0d9787f1d769a0"}
Jul 24 20:23:50.629107 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.629040    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:23:50.636371 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.636346    2566 scope.go:117] "RemoveContainer" containerID="41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524"
Jul 24 20:23:50.636657 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:23:50.636637    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524\": container with ID starting with 41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524 not found: ID does not exist" containerID="41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524"
Jul 24 20:23:50.636714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.636664    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524"} err="failed to get container status \"41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524\": rpc error: code = NotFound desc = could not find container \"41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524\": container with ID starting with 41ba068040babb62c57e15dc718461415a1611f844a8a65cc84d84c617767524 not found: ID does not exist"
Jul 24 20:23:50.664660 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.664612    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb" podStartSLOduration=1.664597134 podStartE2EDuration="1.664597134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:23:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:23:50.664449666 +0000 UTC m=+505.590588573" watchObservedRunningTime="2026-07-24 20:23:50.664597134 +0000 UTC m=+505.590736039"
Jul 24 20:23:50.700599 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.700568    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"]
Jul 24 20:23:50.704913 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:50.704867    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-8684c9ff49-vsmzd"]
Jul 24 20:23:51.692020 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:23:51.691982    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="588e9d0d-0551-4fdc-8f1e-4892dff72a04" path="/var/lib/kubelet/pods/588e9d0d-0551-4fdc-8f1e-4892dff72a04/volumes"
Jul 24 20:24:01.637601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:01.637563    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:24:01.685968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:01.685935    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"]
Jul 24 20:24:01.686163 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:01.686128    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv" podUID="6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" containerName="manager" containerID="cri-o://6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0" gracePeriod=10
Jul 24 20:24:01.934436 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:01.934411    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:24:02.024087 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.024049    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wtjnq\" (UniqueName: \"kubernetes.io/projected/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-kube-api-access-wtjnq\") pod \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") "
Jul 24 20:24:02.024279 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.024163    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-extensions-socket-volume\") pod \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\" (UID: \"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1\") "
Jul 24 20:24:02.024549 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.024508    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" (UID: "6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:24:02.026201 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.026177    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-kube-api-access-wtjnq" (OuterVolumeSpecName: "kube-api-access-wtjnq") pod "6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" (UID: "6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1"). InnerVolumeSpecName "kube-api-access-wtjnq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:24:02.125234 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.125196    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wtjnq\" (UniqueName: \"kubernetes.io/projected/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-kube-api-access-wtjnq\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:02.125234 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.125225    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1-extensions-socket-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:02.690667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.690630    2566 generic.go:356] "Generic (PLEG): container finished" podID="6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" containerID="6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0" exitCode=0
Jul 24 20:24:02.691113 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.690694    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"
Jul 24 20:24:02.691113 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.690696    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv" event={"ID":"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1","Type":"ContainerDied","Data":"6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0"}
Jul 24 20:24:02.691113 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.690815    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv" event={"ID":"6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1","Type":"ContainerDied","Data":"7cebc82efc4d846fd60f08db774672fa7afb8efbdd4758939522a670c630825f"}
Jul 24 20:24:02.691113 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.690854    2566 scope.go:117] "RemoveContainer" containerID="6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0"
Jul 24 20:24:02.699655 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.699640    2566 scope.go:117] "RemoveContainer" containerID="6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0"
Jul 24 20:24:02.699894 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:24:02.699875    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0\": container with ID starting with 6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0 not found: ID does not exist" containerID="6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0"
Jul 24 20:24:02.699969 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.699909    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0"} err="failed to get container status \"6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0\": rpc error: code = NotFound desc = could not find container \"6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0\": container with ID starting with 6cb516f64e2abf9fe5881cf019ff6389d70bb0f7c56eb3064fb83cc7deac2db0 not found: ID does not exist"
Jul 24 20:24:02.712538 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.712510    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"]
Jul 24 20:24:02.718310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:02.718286    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-6fb7879bc9-spkqv"]
Jul 24 20:24:03.692439 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:03.692403    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" path="/var/lib/kubelet/pods/6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1/volumes"
Jul 24 20:24:06.032712 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.032681    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"]
Jul 24 20:24:06.033128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.033032    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" containerName="manager"
Jul 24 20:24:06.033128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.033044    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" containerName="manager"
Jul 24 20:24:06.033128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.033052    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="588e9d0d-0551-4fdc-8f1e-4892dff72a04" containerName="manager"
Jul 24 20:24:06.033128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.033057    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="588e9d0d-0551-4fdc-8f1e-4892dff72a04" containerName="manager"
Jul 24 20:24:06.033128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.033100    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="6134c8b6-3cf3-447b-8a90-cedfa0d7a9b1" containerName="manager"
Jul 24 20:24:06.033128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.033108    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="588e9d0d-0551-4fdc-8f1e-4892dff72a04" containerName="manager"
Jul 24 20:24:06.035511 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.035495    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.038418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.038400    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-6ktdx\""
Jul 24 20:24:06.049325 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.049303    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"]
Jul 24 20:24:06.158392 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158595 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158403    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7372b3-b090-433d-8e7e-f7d6299344ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158595 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158457    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158595 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158478    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158595 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd9hf\" (UniqueName: \"kubernetes.io/projected/3b7372b3-b090-433d-8e7e-f7d6299344ed-kube-api-access-kd9hf\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158796 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158596    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158796 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158796 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158651    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.158796 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.158710    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259273 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259238    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259461 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259461 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259309    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259461 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259443    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259666 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259519    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259666 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259596    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7372b3-b090-433d-8e7e-f7d6299344ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259769 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259769 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259769 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259724    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd9hf\" (UniqueName: \"kubernetes.io/projected/3b7372b3-b090-433d-8e7e-f7d6299344ed-kube-api-access-kd9hf\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259913 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259790    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-workload-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259913 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259805    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-data\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.259913 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259886    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-credential-socket\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.260052 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.259963    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-workload-certs\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.260320 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.260300    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3b7372b3-b090-433d-8e7e-f7d6299344ed-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.261822 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.261801    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-envoy\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.262089 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.262069    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-podinfo\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.268982 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.268956    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3b7372b3-b090-433d-8e7e-f7d6299344ed-istio-token\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.269158 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.269141    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd9hf\" (UniqueName: \"kubernetes.io/projected/3b7372b3-b090-433d-8e7e-f7d6299344ed-kube-api-access-kd9hf\") pod \"maas-default-gateway-openshift-default-78bb78ff78-kpw6c\" (UID: \"3b7372b3-b090-433d-8e7e-f7d6299344ed\") " pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.346232 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.346134    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:06.480217 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.480137    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"]
Jul 24 20:24:06.483122 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:06.483097    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b7372b3_b090_433d_8e7e_f7d6299344ed.slice/crio-bbd03bec799e4a596a1506a9af465389aa356e9fef4788d22492acf3f44711b2 WatchSource:0}: Error finding container bbd03bec799e4a596a1506a9af465389aa356e9fef4788d22492acf3f44711b2: Status 404 returned error can't find the container with id bbd03bec799e4a596a1506a9af465389aa356e9fef4788d22492acf3f44711b2
Jul 24 20:24:06.485959 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.485924    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:24:06.486036 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.486001    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:24:06.486075 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.486043    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:24:06.709601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.709567    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" event={"ID":"3b7372b3-b090-433d-8e7e-f7d6299344ed","Type":"ContainerStarted","Data":"9b0277ed0e756a9a85e932aa2cd96151e8255bd797e479a2e758cb7eb67387ba"}
Jul 24 20:24:06.709780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.709622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" event={"ID":"3b7372b3-b090-433d-8e7e-f7d6299344ed","Type":"ContainerStarted","Data":"bbd03bec799e4a596a1506a9af465389aa356e9fef4788d22492acf3f44711b2"}
Jul 24 20:24:06.729088 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:06.729033    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" podStartSLOduration=0.729018515 podStartE2EDuration="729.018515ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:06.726948758 +0000 UTC m=+521.653087663" watchObservedRunningTime="2026-07-24 20:24:06.729018515 +0000 UTC m=+521.655157465"
Jul 24 20:24:07.346334 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:07.346298    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:07.351281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:07.351252    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:07.713754 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:07.713728    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:07.714617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:07.714595    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:24:19.804842 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.804770    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-46cqk"]
Jul 24 20:24:19.806901 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.806885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:19.809929 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.809900    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-9vz6v\""
Jul 24 20:24:19.810041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.809947    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Jul 24 20:24:19.818075 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.818055    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-46cqk"]
Jul 24 20:24:19.897784 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.897756    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-46cqk"]
Jul 24 20:24:19.961047 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.961019    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-config-file\") pod \"limitador-limitador-64c94fd64f-46cqk\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:19.961168 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:19.961063    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4g55\" (UniqueName: \"kubernetes.io/projected/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-kube-api-access-z4g55\") pod \"limitador-limitador-64c94fd64f-46cqk\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:20.062101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.062025    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4g55\" (UniqueName: \"kubernetes.io/projected/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-kube-api-access-z4g55\") pod \"limitador-limitador-64c94fd64f-46cqk\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:20.062101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.062088    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-config-file\") pod \"limitador-limitador-64c94fd64f-46cqk\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:20.062690 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.062673    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-config-file\") pod \"limitador-limitador-64c94fd64f-46cqk\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:20.071191 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.071171    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4g55\" (UniqueName: \"kubernetes.io/projected/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-kube-api-access-z4g55\") pod \"limitador-limitador-64c94fd64f-46cqk\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") " pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:20.117205 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.117181    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:20.243642 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.243511    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-46cqk"]
Jul 24 20:24:20.246058 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:20.246032    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode773c7b5_e628_4a1b_ae0c_1e2638df52ca.slice/crio-79a7347de3874ee882789263b42d34c71cd8dd4b3a0fe19c97ca8e4d0237a3ea WatchSource:0}: Error finding container 79a7347de3874ee882789263b42d34c71cd8dd4b3a0fe19c97ca8e4d0237a3ea: Status 404 returned error can't find the container with id 79a7347de3874ee882789263b42d34c71cd8dd4b3a0fe19c97ca8e4d0237a3ea
Jul 24 20:24:20.582101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.582067    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-58cd955675-qtp5l"]
Jul 24 20:24:20.585233 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.585214    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtp5l"
Jul 24 20:24:20.587979 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.587853    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-5gsf6\""
Jul 24 20:24:20.592829 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.592782    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtp5l"]
Jul 24 20:24:20.665636 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.665610    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vmfn7\" (UniqueName: \"kubernetes.io/projected/da66110b-7c09-4c74-8720-7b3db83985ca-kube-api-access-vmfn7\") pod \"authorino-58cd955675-qtp5l\" (UID: \"da66110b-7c09-4c74-8720-7b3db83985ca\") " pod="kuadrant-system/authorino-58cd955675-qtp5l"
Jul 24 20:24:20.757632 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.757604    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg24g"]
Jul 24 20:24:20.759747 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.759729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg24g"
Jul 24 20:24:20.766139 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.765991    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vmfn7\" (UniqueName: \"kubernetes.io/projected/da66110b-7c09-4c74-8720-7b3db83985ca-kube-api-access-vmfn7\") pod \"authorino-58cd955675-qtp5l\" (UID: \"da66110b-7c09-4c74-8720-7b3db83985ca\") " pod="kuadrant-system/authorino-58cd955675-qtp5l"
Jul 24 20:24:20.767519 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.767496    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg24g"]
Jul 24 20:24:20.770658 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.770628    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk" event={"ID":"e773c7b5-e628-4a1b-ae0c-1e2638df52ca","Type":"ContainerStarted","Data":"79a7347de3874ee882789263b42d34c71cd8dd4b3a0fe19c97ca8e4d0237a3ea"}
Jul 24 20:24:20.782730 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.780143    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vmfn7\" (UniqueName: \"kubernetes.io/projected/da66110b-7c09-4c74-8720-7b3db83985ca-kube-api-access-vmfn7\") pod \"authorino-58cd955675-qtp5l\" (UID: \"da66110b-7c09-4c74-8720-7b3db83985ca\") " pod="kuadrant-system/authorino-58cd955675-qtp5l"
Jul 24 20:24:20.866437 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.866412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7kqh\" (UniqueName: \"kubernetes.io/projected/defce69c-7195-4747-8d8c-56ce1d17caf9-kube-api-access-c7kqh\") pod \"authorino-75d556bd77-gg24g\" (UID: \"defce69c-7195-4747-8d8c-56ce1d17caf9\") " pod="kuadrant-system/authorino-75d556bd77-gg24g"
Jul 24 20:24:20.895396 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.895374    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtp5l"
Jul 24 20:24:20.967787 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.967755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7kqh\" (UniqueName: \"kubernetes.io/projected/defce69c-7195-4747-8d8c-56ce1d17caf9-kube-api-access-c7kqh\") pod \"authorino-75d556bd77-gg24g\" (UID: \"defce69c-7195-4747-8d8c-56ce1d17caf9\") " pod="kuadrant-system/authorino-75d556bd77-gg24g"
Jul 24 20:24:20.977643 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:20.977590    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7kqh\" (UniqueName: \"kubernetes.io/projected/defce69c-7195-4747-8d8c-56ce1d17caf9-kube-api-access-c7kqh\") pod \"authorino-75d556bd77-gg24g\" (UID: \"defce69c-7195-4747-8d8c-56ce1d17caf9\") " pod="kuadrant-system/authorino-75d556bd77-gg24g"
Jul 24 20:24:21.037465 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:21.037398    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtp5l"]
Jul 24 20:24:21.039058 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:21.039023    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podda66110b_7c09_4c74_8720_7b3db83985ca.slice/crio-0feb3307f320e7ba95c7851ac37aa66b6df792244b9238f2d5b084a8df049ed1 WatchSource:0}: Error finding container 0feb3307f320e7ba95c7851ac37aa66b6df792244b9238f2d5b084a8df049ed1: Status 404 returned error can't find the container with id 0feb3307f320e7ba95c7851ac37aa66b6df792244b9238f2d5b084a8df049ed1
Jul 24 20:24:21.068447 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:21.068421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg24g"
Jul 24 20:24:21.235372 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:21.235254    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg24g"]
Jul 24 20:24:21.244154 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:21.244122    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddefce69c_7195_4747_8d8c_56ce1d17caf9.slice/crio-b57021017af1b39986693c8b7cfdcf1ae229b582287c5d63771e209495e62702 WatchSource:0}: Error finding container b57021017af1b39986693c8b7cfdcf1ae229b582287c5d63771e209495e62702: Status 404 returned error can't find the container with id b57021017af1b39986693c8b7cfdcf1ae229b582287c5d63771e209495e62702
Jul 24 20:24:21.779328 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:21.779287    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtp5l" event={"ID":"da66110b-7c09-4c74-8720-7b3db83985ca","Type":"ContainerStarted","Data":"0feb3307f320e7ba95c7851ac37aa66b6df792244b9238f2d5b084a8df049ed1"}
Jul 24 20:24:21.781165 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:21.781132    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg24g" event={"ID":"defce69c-7195-4747-8d8c-56ce1d17caf9","Type":"ContainerStarted","Data":"b57021017af1b39986693c8b7cfdcf1ae229b582287c5d63771e209495e62702"}
Jul 24 20:24:25.800617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.800516    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtp5l" event={"ID":"da66110b-7c09-4c74-8720-7b3db83985ca","Type":"ContainerStarted","Data":"3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c"}
Jul 24 20:24:25.801949 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.801920    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk" event={"ID":"e773c7b5-e628-4a1b-ae0c-1e2638df52ca","Type":"ContainerStarted","Data":"71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158"}
Jul 24 20:24:25.802066 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.802020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:25.803352 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.803334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg24g" event={"ID":"defce69c-7195-4747-8d8c-56ce1d17caf9","Type":"ContainerStarted","Data":"0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c"}
Jul 24 20:24:25.816512 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.816472    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-58cd955675-qtp5l" podStartSLOduration=1.966127803 podStartE2EDuration="5.81645903s" totalImagesPullingTime="3.850331227s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:25.814966382 +0000 UTC m=+540.741105288" watchObservedRunningTime="2026-07-24 20:24:25.81645903 +0000 UTC m=+540.742597935"
Jul 24 20:24:25.833238 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.833199    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk" podStartSLOduration=2.195798732 podStartE2EDuration="6.833187159s" totalImagesPullingTime="4.637388427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:25.832166208 +0000 UTC m=+540.758305112" watchObservedRunningTime="2026-07-24 20:24:25.833187159 +0000 UTC m=+540.759326061"
Jul 24 20:24:25.847978 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.847930    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-75d556bd77-gg24g" podStartSLOduration=2.204949798 podStartE2EDuration="5.847920344s" totalImagesPullingTime="3.642970546s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:25.84631344 +0000 UTC m=+540.772452350" watchObservedRunningTime="2026-07-24 20:24:25.847920344 +0000 UTC m=+540.774059248"
Jul 24 20:24:25.868049 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:25.868023    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtp5l"]
Jul 24 20:24:27.811737 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:27.811694    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-58cd955675-qtp5l" podUID="da66110b-7c09-4c74-8720-7b3db83985ca" containerName="authorino" containerID="cri-o://3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c" gracePeriod=30
Jul 24 20:24:28.046474 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.046446    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtp5l"
Jul 24 20:24:28.127076 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.127053    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vmfn7\" (UniqueName: \"kubernetes.io/projected/da66110b-7c09-4c74-8720-7b3db83985ca-kube-api-access-vmfn7\") pod \"da66110b-7c09-4c74-8720-7b3db83985ca\" (UID: \"da66110b-7c09-4c74-8720-7b3db83985ca\") "
Jul 24 20:24:28.128880 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.128855    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da66110b-7c09-4c74-8720-7b3db83985ca-kube-api-access-vmfn7" (OuterVolumeSpecName: "kube-api-access-vmfn7") pod "da66110b-7c09-4c74-8720-7b3db83985ca" (UID: "da66110b-7c09-4c74-8720-7b3db83985ca"). InnerVolumeSpecName "kube-api-access-vmfn7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:24:28.228173 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.228150    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vmfn7\" (UniqueName: \"kubernetes.io/projected/da66110b-7c09-4c74-8720-7b3db83985ca-kube-api-access-vmfn7\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:28.816443 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.816410    2566 generic.go:356] "Generic (PLEG): container finished" podID="da66110b-7c09-4c74-8720-7b3db83985ca" containerID="3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c" exitCode=0
Jul 24 20:24:28.816876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.816451    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-58cd955675-qtp5l"
Jul 24 20:24:28.816876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.816473    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtp5l" event={"ID":"da66110b-7c09-4c74-8720-7b3db83985ca","Type":"ContainerDied","Data":"3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c"}
Jul 24 20:24:28.816876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.816503    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-58cd955675-qtp5l" event={"ID":"da66110b-7c09-4c74-8720-7b3db83985ca","Type":"ContainerDied","Data":"0feb3307f320e7ba95c7851ac37aa66b6df792244b9238f2d5b084a8df049ed1"}
Jul 24 20:24:28.816876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.816520    2566 scope.go:117] "RemoveContainer" containerID="3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c"
Jul 24 20:24:28.824568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.824551    2566 scope.go:117] "RemoveContainer" containerID="3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c"
Jul 24 20:24:28.824813 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:24:28.824796    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c\": container with ID starting with 3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c not found: ID does not exist" containerID="3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c"
Jul 24 20:24:28.824858 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.824820    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c"} err="failed to get container status \"3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c\": rpc error: code = NotFound desc = could not find container \"3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c\": container with ID starting with 3d07d85c6e37745fb78646fd484beafe1b741972d058c2c985050ac8942f964c not found: ID does not exist"
Jul 24 20:24:28.836210 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.836190    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtp5l"]
Jul 24 20:24:28.841396 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:28.841377    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-58cd955675-qtp5l"]
Jul 24 20:24:29.697717 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:29.697682    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="da66110b-7c09-4c74-8720-7b3db83985ca" path="/var/lib/kubelet/pods/da66110b-7c09-4c74-8720-7b3db83985ca/volumes"
Jul 24 20:24:34.567506 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:34.567470    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-46cqk"]
Jul 24 20:24:34.568003 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:34.567682    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk" podUID="e773c7b5-e628-4a1b-ae0c-1e2638df52ca" containerName="limitador" containerID="cri-o://71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158" gracePeriod=30
Jul 24 20:24:34.568569 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:34.568454    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:35.116279 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.116257    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:35.276098 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.276031    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-config-file\") pod \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") "
Jul 24 20:24:35.276243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.276102    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z4g55\" (UniqueName: \"kubernetes.io/projected/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-kube-api-access-z4g55\") pod \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\" (UID: \"e773c7b5-e628-4a1b-ae0c-1e2638df52ca\") "
Jul 24 20:24:35.276359 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.276337    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-config-file" (OuterVolumeSpecName: "config-file") pod "e773c7b5-e628-4a1b-ae0c-1e2638df52ca" (UID: "e773c7b5-e628-4a1b-ae0c-1e2638df52ca"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:24:35.278066 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.278045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-kube-api-access-z4g55" (OuterVolumeSpecName: "kube-api-access-z4g55") pod "e773c7b5-e628-4a1b-ae0c-1e2638df52ca" (UID: "e773c7b5-e628-4a1b-ae0c-1e2638df52ca"). InnerVolumeSpecName "kube-api-access-z4g55". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:24:35.377487 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.377459    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4g55\" (UniqueName: \"kubernetes.io/projected/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-kube-api-access-z4g55\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:35.377487 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.377482    2566 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/e773c7b5-e628-4a1b-ae0c-1e2638df52ca-config-file\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:35.844899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.844868    2566 generic.go:356] "Generic (PLEG): container finished" podID="e773c7b5-e628-4a1b-ae0c-1e2638df52ca" containerID="71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158" exitCode=0
Jul 24 20:24:35.845300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.844919    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk"
Jul 24 20:24:35.845300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.844929    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk" event={"ID":"e773c7b5-e628-4a1b-ae0c-1e2638df52ca","Type":"ContainerDied","Data":"71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158"}
Jul 24 20:24:35.845300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.844967    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-64c94fd64f-46cqk" event={"ID":"e773c7b5-e628-4a1b-ae0c-1e2638df52ca","Type":"ContainerDied","Data":"79a7347de3874ee882789263b42d34c71cd8dd4b3a0fe19c97ca8e4d0237a3ea"}
Jul 24 20:24:35.845300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.844989    2566 scope.go:117] "RemoveContainer" containerID="71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158"
Jul 24 20:24:35.853257 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.853242    2566 scope.go:117] "RemoveContainer" containerID="71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158"
Jul 24 20:24:35.853487 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:24:35.853472    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158\": container with ID starting with 71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158 not found: ID does not exist" containerID="71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158"
Jul 24 20:24:35.853562 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.853497    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158"} err="failed to get container status \"71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158\": rpc error: code = NotFound desc = could not find container \"71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158\": container with ID starting with 71df990a4a3583a69728ab66c5e57a1cab5891ffebd45b72b4425d37bb5e0158 not found: ID does not exist"
Jul 24 20:24:35.860156 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.860136    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-46cqk"]
Jul 24 20:24:35.863446 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:35.863427    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-64c94fd64f-46cqk"]
Jul 24 20:24:37.691287 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:37.691252    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e773c7b5-e628-4a1b-ae0c-1e2638df52ca" path="/var/lib/kubelet/pods/e773c7b5-e628-4a1b-ae0c-1e2638df52ca/volumes"
Jul 24 20:24:39.706233 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.706201    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zngps"]
Jul 24 20:24:39.706645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.706513    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e773c7b5-e628-4a1b-ae0c-1e2638df52ca" containerName="limitador"
Jul 24 20:24:39.706645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.706548    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e773c7b5-e628-4a1b-ae0c-1e2638df52ca" containerName="limitador"
Jul 24 20:24:39.706645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.706566    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="da66110b-7c09-4c74-8720-7b3db83985ca" containerName="authorino"
Jul 24 20:24:39.706645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.706571    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="da66110b-7c09-4c74-8720-7b3db83985ca" containerName="authorino"
Jul 24 20:24:39.706645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.706624    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="da66110b-7c09-4c74-8720-7b3db83985ca" containerName="authorino"
Jul 24 20:24:39.706645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.706633    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e773c7b5-e628-4a1b-ae0c-1e2638df52ca" containerName="limitador"
Jul 24 20:24:39.709140 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.709123    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:39.711801 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.711773    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\""
Jul 24 20:24:39.712027 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.712004    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Jul 24 20:24:39.712102 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.712007    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Jul 24 20:24:39.712980 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.712960    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-jkvdb\""
Jul 24 20:24:39.724774 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.724750    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zngps"]
Jul 24 20:24:39.805762 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.805738    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/13bfd465-727d-4bac-ba4d-217febf38ddf-data\") pod \"postgres-69fb4594cf-zngps\" (UID: \"13bfd465-727d-4bac-ba4d-217febf38ddf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:39.805890 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.805858    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk952\" (UniqueName: \"kubernetes.io/projected/13bfd465-727d-4bac-ba4d-217febf38ddf-kube-api-access-sk952\") pod \"postgres-69fb4594cf-zngps\" (UID: \"13bfd465-727d-4bac-ba4d-217febf38ddf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:39.907070 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.907044    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk952\" (UniqueName: \"kubernetes.io/projected/13bfd465-727d-4bac-ba4d-217febf38ddf-kube-api-access-sk952\") pod \"postgres-69fb4594cf-zngps\" (UID: \"13bfd465-727d-4bac-ba4d-217febf38ddf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:39.907180 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.907083    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/13bfd465-727d-4bac-ba4d-217febf38ddf-data\") pod \"postgres-69fb4594cf-zngps\" (UID: \"13bfd465-727d-4bac-ba4d-217febf38ddf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:39.907378 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.907364    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/13bfd465-727d-4bac-ba4d-217febf38ddf-data\") pod \"postgres-69fb4594cf-zngps\" (UID: \"13bfd465-727d-4bac-ba4d-217febf38ddf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:39.915471 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:39.915453    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk952\" (UniqueName: \"kubernetes.io/projected/13bfd465-727d-4bac-ba4d-217febf38ddf-kube-api-access-sk952\") pod \"postgres-69fb4594cf-zngps\" (UID: \"13bfd465-727d-4bac-ba4d-217febf38ddf\") " pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:40.021140 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:40.021079    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:40.145231 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:40.145207    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-69fb4594cf-zngps"]
Jul 24 20:24:40.148164 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:40.148131    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bfd465_727d_4bac_ba4d_217febf38ddf.slice/crio-f257adc3612fbc573a8c4e1ff08ef44bf4bbd7b249b9f53ce143eb09603f1625 WatchSource:0}: Error finding container f257adc3612fbc573a8c4e1ff08ef44bf4bbd7b249b9f53ce143eb09603f1625: Status 404 returned error can't find the container with id f257adc3612fbc573a8c4e1ff08ef44bf4bbd7b249b9f53ce143eb09603f1625
Jul 24 20:24:40.866521 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:40.866488    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps" event={"ID":"13bfd465-727d-4bac-ba4d-217febf38ddf","Type":"ContainerStarted","Data":"f257adc3612fbc573a8c4e1ff08ef44bf4bbd7b249b9f53ce143eb09603f1625"}
Jul 24 20:24:45.891304 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:45.891266    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps" event={"ID":"13bfd465-727d-4bac-ba4d-217febf38ddf","Type":"ContainerStarted","Data":"ac26c21ae86bb7fa10a7070ad272841c06deaec65fe0b6b38f795f25cf44ae79"}
Jul 24 20:24:45.891724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:45.891401    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:45.911608 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:45.911561    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps" podStartSLOduration=1.9090735140000001 podStartE2EDuration="6.91154935s" totalImagesPullingTime="5.002475836s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:45.909655828 +0000 UTC m=+560.835794735" watchObservedRunningTime="2026-07-24 20:24:45.91154935 +0000 UTC m=+560.837688256"
Jul 24 20:24:51.926829 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:51.926799    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-69fb4594cf-zngps"
Jul 24 20:24:52.489652 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.489623    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-5d4b67c446-b48dv"]
Jul 24 20:24:52.492805 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.492784    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-b48dv"
Jul 24 20:24:52.493012 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.492990    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-b48dv"]
Jul 24 20:24:52.605448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.605424    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m964\" (UniqueName: \"kubernetes.io/projected/55d415d9-52a7-4947-981d-1ef991a902fe-kube-api-access-8m964\") pod \"authorino-5d4b67c446-b48dv\" (UID: \"55d415d9-52a7-4947-981d-1ef991a902fe\") " pod="kuadrant-system/authorino-5d4b67c446-b48dv"
Jul 24 20:24:52.706797 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.706774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m964\" (UniqueName: \"kubernetes.io/projected/55d415d9-52a7-4947-981d-1ef991a902fe-kube-api-access-8m964\") pod \"authorino-5d4b67c446-b48dv\" (UID: \"55d415d9-52a7-4947-981d-1ef991a902fe\") " pod="kuadrant-system/authorino-5d4b67c446-b48dv"
Jul 24 20:24:52.715402 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.715374    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m964\" (UniqueName: \"kubernetes.io/projected/55d415d9-52a7-4947-981d-1ef991a902fe-kube-api-access-8m964\") pod \"authorino-5d4b67c446-b48dv\" (UID: \"55d415d9-52a7-4947-981d-1ef991a902fe\") " pod="kuadrant-system/authorino-5d4b67c446-b48dv"
Jul 24 20:24:52.728377 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.728351    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-b48dv"]
Jul 24 20:24:52.728563 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.728550    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-b48dv"
Jul 24 20:24:52.755488 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.755465    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7846cd655c-gfbwp"]
Jul 24 20:24:52.760657 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.760638    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7846cd655c-gfbwp"
Jul 24 20:24:52.767794 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.767772    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7846cd655c-gfbwp"]
Jul 24 20:24:52.863262 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.863227    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-b48dv"]
Jul 24 20:24:52.867583 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:52.867557    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod55d415d9_52a7_4947_981d_1ef991a902fe.slice/crio-8324a901c88c86238d68661c103d698b6eef2f8ba7c732089a72e47415b406a6 WatchSource:0}: Error finding container 8324a901c88c86238d68661c103d698b6eef2f8ba7c732089a72e47415b406a6: Status 404 returned error can't find the container with id 8324a901c88c86238d68661c103d698b6eef2f8ba7c732089a72e47415b406a6
Jul 24 20:24:52.908730 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.908706    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46gmb\" (UniqueName: \"kubernetes.io/projected/46970387-be02-4bab-ace6-944b14e17aec-kube-api-access-46gmb\") pod \"authorino-7846cd655c-gfbwp\" (UID: \"46970387-be02-4bab-ace6-944b14e17aec\") " pod="kuadrant-system/authorino-7846cd655c-gfbwp"
Jul 24 20:24:52.924324 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:52.924296    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-b48dv" event={"ID":"55d415d9-52a7-4947-981d-1ef991a902fe","Type":"ContainerStarted","Data":"8324a901c88c86238d68661c103d698b6eef2f8ba7c732089a72e47415b406a6"}
Jul 24 20:24:53.009602 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.009548    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46gmb\" (UniqueName: \"kubernetes.io/projected/46970387-be02-4bab-ace6-944b14e17aec-kube-api-access-46gmb\") pod \"authorino-7846cd655c-gfbwp\" (UID: \"46970387-be02-4bab-ace6-944b14e17aec\") " pod="kuadrant-system/authorino-7846cd655c-gfbwp"
Jul 24 20:24:53.019904 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.019884    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46gmb\" (UniqueName: \"kubernetes.io/projected/46970387-be02-4bab-ace6-944b14e17aec-kube-api-access-46gmb\") pod \"authorino-7846cd655c-gfbwp\" (UID: \"46970387-be02-4bab-ace6-944b14e17aec\") " pod="kuadrant-system/authorino-7846cd655c-gfbwp"
Jul 24 20:24:53.052750 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.052727    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7846cd655c-gfbwp"]
Jul 24 20:24:53.052917 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.052906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7846cd655c-gfbwp"
Jul 24 20:24:53.079760 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.079738    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59f6fcf49c-x6brm"]
Jul 24 20:24:53.119468 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.119446    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59f6fcf49c-x6brm"]
Jul 24 20:24:53.119656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.119580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.122359 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.122339    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Jul 24 20:24:53.194646 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.194623    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7846cd655c-gfbwp"]
Jul 24 20:24:53.196434 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:53.196411    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46970387_be02_4bab_ace6_944b14e17aec.slice/crio-75861308659c0894c42c1ffac9264c092daf6555db557e8204acab4773bbec68 WatchSource:0}: Error finding container 75861308659c0894c42c1ffac9264c092daf6555db557e8204acab4773bbec68: Status 404 returned error can't find the container with id 75861308659c0894c42c1ffac9264c092daf6555db557e8204acab4773bbec68
Jul 24 20:24:53.311429 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.311370    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baeb603a-ff5c-4996-8456-f39138c5a550-tls-cert\") pod \"authorino-59f6fcf49c-x6brm\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") " pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.311429 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.311412    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jf9q7\" (UniqueName: \"kubernetes.io/projected/baeb603a-ff5c-4996-8456-f39138c5a550-kube-api-access-jf9q7\") pod \"authorino-59f6fcf49c-x6brm\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") " pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.411959 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.411936    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baeb603a-ff5c-4996-8456-f39138c5a550-tls-cert\") pod \"authorino-59f6fcf49c-x6brm\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") " pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.412042 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.411976    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jf9q7\" (UniqueName: \"kubernetes.io/projected/baeb603a-ff5c-4996-8456-f39138c5a550-kube-api-access-jf9q7\") pod \"authorino-59f6fcf49c-x6brm\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") " pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.413995 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.413972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baeb603a-ff5c-4996-8456-f39138c5a550-tls-cert\") pod \"authorino-59f6fcf49c-x6brm\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") " pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.419796 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.419775    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jf9q7\" (UniqueName: \"kubernetes.io/projected/baeb603a-ff5c-4996-8456-f39138c5a550-kube-api-access-jf9q7\") pod \"authorino-59f6fcf49c-x6brm\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") " pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.430560 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.430516    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:24:53.576753 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.576686    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59f6fcf49c-x6brm"]
Jul 24 20:24:53.579446 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:24:53.579420    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podbaeb603a_ff5c_4996_8456_f39138c5a550.slice/crio-10138c61504a86d88649daf2ac26760a6c01d7303f0418db348c2b662b0ba10c WatchSource:0}: Error finding container 10138c61504a86d88649daf2ac26760a6c01d7303f0418db348c2b662b0ba10c: Status 404 returned error can't find the container with id 10138c61504a86d88649daf2ac26760a6c01d7303f0418db348c2b662b0ba10c
Jul 24 20:24:53.930922 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.930876    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-b48dv" event={"ID":"55d415d9-52a7-4947-981d-1ef991a902fe","Type":"ContainerStarted","Data":"40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6"}
Jul 24 20:24:53.931097 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.931070    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-5d4b67c446-b48dv" podUID="55d415d9-52a7-4947-981d-1ef991a902fe" containerName="authorino" containerID="cri-o://40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6" gracePeriod=30
Jul 24 20:24:53.933238 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.933211    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f6fcf49c-x6brm" event={"ID":"baeb603a-ff5c-4996-8456-f39138c5a550","Type":"ContainerStarted","Data":"10138c61504a86d88649daf2ac26760a6c01d7303f0418db348c2b662b0ba10c"}
Jul 24 20:24:53.935154 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.935126    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7846cd655c-gfbwp" event={"ID":"46970387-be02-4bab-ace6-944b14e17aec","Type":"ContainerStarted","Data":"0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e"}
Jul 24 20:24:53.935241 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.935162    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7846cd655c-gfbwp" podUID="46970387-be02-4bab-ace6-944b14e17aec" containerName="authorino" containerID="cri-o://0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e" gracePeriod=30
Jul 24 20:24:53.935305 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.935167    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7846cd655c-gfbwp" event={"ID":"46970387-be02-4bab-ace6-944b14e17aec","Type":"ContainerStarted","Data":"75861308659c0894c42c1ffac9264c092daf6555db557e8204acab4773bbec68"}
Jul 24 20:24:53.946850 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.946811    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-5d4b67c446-b48dv" podStartSLOduration=1.4708465020000001 podStartE2EDuration="1.946794832s" totalImagesPullingTime="475.94833ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:53.945588408 +0000 UTC m=+568.871727314" watchObservedRunningTime="2026-07-24 20:24:53.946794832 +0000 UTC m=+568.872933742"
Jul 24 20:24:53.960589 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:53.960552    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7846cd655c-gfbwp" podStartSLOduration=1.38823728 podStartE2EDuration="1.960540474s" totalImagesPullingTime="572.303194ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:53.959638376 +0000 UTC m=+568.885777282" watchObservedRunningTime="2026-07-24 20:24:53.960540474 +0000 UTC m=+568.886679370"
Jul 24 20:24:54.216497 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.216468    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-b48dv"
Jul 24 20:24:54.231028 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.231001    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7846cd655c-gfbwp"
Jul 24 20:24:54.319314 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.319291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8m964\" (UniqueName: \"kubernetes.io/projected/55d415d9-52a7-4947-981d-1ef991a902fe-kube-api-access-8m964\") pod \"55d415d9-52a7-4947-981d-1ef991a902fe\" (UID: \"55d415d9-52a7-4947-981d-1ef991a902fe\") "
Jul 24 20:24:54.321190 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.321167    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d415d9-52a7-4947-981d-1ef991a902fe-kube-api-access-8m964" (OuterVolumeSpecName: "kube-api-access-8m964") pod "55d415d9-52a7-4947-981d-1ef991a902fe" (UID: "55d415d9-52a7-4947-981d-1ef991a902fe"). InnerVolumeSpecName "kube-api-access-8m964". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:24:54.420748 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.420723    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46gmb\" (UniqueName: \"kubernetes.io/projected/46970387-be02-4bab-ace6-944b14e17aec-kube-api-access-46gmb\") pod \"46970387-be02-4bab-ace6-944b14e17aec\" (UID: \"46970387-be02-4bab-ace6-944b14e17aec\") "
Jul 24 20:24:54.420933 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.420918    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m964\" (UniqueName: \"kubernetes.io/projected/55d415d9-52a7-4947-981d-1ef991a902fe-kube-api-access-8m964\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:54.422577 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.422552    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46970387-be02-4bab-ace6-944b14e17aec-kube-api-access-46gmb" (OuterVolumeSpecName: "kube-api-access-46gmb") pod "46970387-be02-4bab-ace6-944b14e17aec" (UID: "46970387-be02-4bab-ace6-944b14e17aec"). InnerVolumeSpecName "kube-api-access-46gmb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:24:54.521450 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.521429    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46gmb\" (UniqueName: \"kubernetes.io/projected/46970387-be02-4bab-ace6-944b14e17aec-kube-api-access-46gmb\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:54.940211 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.940180    2566 generic.go:356] "Generic (PLEG): container finished" podID="55d415d9-52a7-4947-981d-1ef991a902fe" containerID="40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6" exitCode=0
Jul 24 20:24:54.940350 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.940212    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-5d4b67c446-b48dv"
Jul 24 20:24:54.940350 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.940261    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-b48dv" event={"ID":"55d415d9-52a7-4947-981d-1ef991a902fe","Type":"ContainerDied","Data":"40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6"}
Jul 24 20:24:54.940350 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.940303    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-5d4b67c446-b48dv" event={"ID":"55d415d9-52a7-4947-981d-1ef991a902fe","Type":"ContainerDied","Data":"8324a901c88c86238d68661c103d698b6eef2f8ba7c732089a72e47415b406a6"}
Jul 24 20:24:54.940350 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.940329    2566 scope.go:117] "RemoveContainer" containerID="40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6"
Jul 24 20:24:54.942336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.942308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f6fcf49c-x6brm" event={"ID":"baeb603a-ff5c-4996-8456-f39138c5a550","Type":"ContainerStarted","Data":"5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353"}
Jul 24 20:24:54.943937 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.943911    2566 generic.go:356] "Generic (PLEG): container finished" podID="46970387-be02-4bab-ace6-944b14e17aec" containerID="0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e" exitCode=0
Jul 24 20:24:54.944050 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.944021    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7846cd655c-gfbwp"
Jul 24 20:24:54.944050 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.944028    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7846cd655c-gfbwp" event={"ID":"46970387-be02-4bab-ace6-944b14e17aec","Type":"ContainerDied","Data":"0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e"}
Jul 24 20:24:54.944165 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.944066    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7846cd655c-gfbwp" event={"ID":"46970387-be02-4bab-ace6-944b14e17aec","Type":"ContainerDied","Data":"75861308659c0894c42c1ffac9264c092daf6555db557e8204acab4773bbec68"}
Jul 24 20:24:54.950829 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.950452    2566 scope.go:117] "RemoveContainer" containerID="40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6"
Jul 24 20:24:54.954814 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:24:54.954730    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6\": container with ID starting with 40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6 not found: ID does not exist" containerID="40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6"
Jul 24 20:24:54.954814 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.954776    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6"} err="failed to get container status \"40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6\": rpc error: code = NotFound desc = could not find container \"40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6\": container with ID starting with 40ea7e337df27713243bbc13d47dd64deddb3a1d314867bff48bf5b3faca8ec6 not found: ID does not exist"
Jul 24 20:24:54.954814 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.954799    2566 scope.go:117] "RemoveContainer" containerID="0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e"
Jul 24 20:24:54.963302 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.963261    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59f6fcf49c-x6brm" podStartSLOduration=1.534148102 podStartE2EDuration="1.963249148s" totalImagesPullingTime="429.101046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:24:54.961814912 +0000 UTC m=+569.887953822" watchObservedRunningTime="2026-07-24 20:24:54.963249148 +0000 UTC m=+569.889388054"
Jul 24 20:24:54.963497 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.963476    2566 scope.go:117] "RemoveContainer" containerID="0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e"
Jul 24 20:24:54.963775 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:24:54.963756    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e\": container with ID starting with 0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e not found: ID does not exist" containerID="0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e"
Jul 24 20:24:54.963841 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.963782    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e"} err="failed to get container status \"0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e\": rpc error: code = NotFound desc = could not find container \"0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e\": container with ID starting with 0ed949cc5900ef642bc47fe350de6d4d41be34f92a51022d6f173df3d98b0e5e not found: ID does not exist"
Jul 24 20:24:54.976921 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.976869    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-b48dv"]
Jul 24 20:24:54.981492 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.981468    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-5d4b67c446-b48dv"]
Jul 24 20:24:54.988309 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.988289    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg24g"]
Jul 24 20:24:54.988488 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.988471    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-75d556bd77-gg24g" podUID="defce69c-7195-4747-8d8c-56ce1d17caf9" containerName="authorino" containerID="cri-o://0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c" gracePeriod=30
Jul 24 20:24:54.993877 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.993854    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7846cd655c-gfbwp"]
Jul 24 20:24:54.999639 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:54.999617    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7846cd655c-gfbwp"]
Jul 24 20:24:55.289130 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.289106    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg24g"
Jul 24 20:24:55.428818 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.428788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c7kqh\" (UniqueName: \"kubernetes.io/projected/defce69c-7195-4747-8d8c-56ce1d17caf9-kube-api-access-c7kqh\") pod \"defce69c-7195-4747-8d8c-56ce1d17caf9\" (UID: \"defce69c-7195-4747-8d8c-56ce1d17caf9\") "
Jul 24 20:24:55.430775 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.430752    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/defce69c-7195-4747-8d8c-56ce1d17caf9-kube-api-access-c7kqh" (OuterVolumeSpecName: "kube-api-access-c7kqh") pod "defce69c-7195-4747-8d8c-56ce1d17caf9" (UID: "defce69c-7195-4747-8d8c-56ce1d17caf9"). InnerVolumeSpecName "kube-api-access-c7kqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:24:55.530184 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.530132    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7kqh\" (UniqueName: \"kubernetes.io/projected/defce69c-7195-4747-8d8c-56ce1d17caf9-kube-api-access-c7kqh\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:24:55.692233 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.692194    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46970387-be02-4bab-ace6-944b14e17aec" path="/var/lib/kubelet/pods/46970387-be02-4bab-ace6-944b14e17aec/volumes"
Jul 24 20:24:55.692598 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.692580    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="55d415d9-52a7-4947-981d-1ef991a902fe" path="/var/lib/kubelet/pods/55d415d9-52a7-4947-981d-1ef991a902fe/volumes"
Jul 24 20:24:55.949213 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.949184    2566 generic.go:356] "Generic (PLEG): container finished" podID="defce69c-7195-4747-8d8c-56ce1d17caf9" containerID="0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c" exitCode=0
Jul 24 20:24:55.949370 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.949223    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-75d556bd77-gg24g"
Jul 24 20:24:55.949370 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.949269    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg24g" event={"ID":"defce69c-7195-4747-8d8c-56ce1d17caf9","Type":"ContainerDied","Data":"0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c"}
Jul 24 20:24:55.949370 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.949311    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-75d556bd77-gg24g" event={"ID":"defce69c-7195-4747-8d8c-56ce1d17caf9","Type":"ContainerDied","Data":"b57021017af1b39986693c8b7cfdcf1ae229b582287c5d63771e209495e62702"}
Jul 24 20:24:55.949370 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.949330    2566 scope.go:117] "RemoveContainer" containerID="0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c"
Jul 24 20:24:55.956918 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.956904    2566 scope.go:117] "RemoveContainer" containerID="0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c"
Jul 24 20:24:55.957158 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:24:55.957141    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c\": container with ID starting with 0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c not found: ID does not exist" containerID="0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c"
Jul 24 20:24:55.957208 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.957169    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c"} err="failed to get container status \"0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c\": rpc error: code = NotFound desc = could not find container \"0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c\": container with ID starting with 0726a03aff1ea2334f520f622654747b95bea28c6e510abad1833f20d39ded4c not found: ID does not exist"
Jul 24 20:24:55.965880 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.965860    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg24g"]
Jul 24 20:24:55.969789 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:55.969769    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-75d556bd77-gg24g"]
Jul 24 20:24:57.692535 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:57.692493    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="defce69c-7195-4747-8d8c-56ce1d17caf9" path="/var/lib/kubelet/pods/defce69c-7195-4747-8d8c-56ce1d17caf9/volumes"
Jul 24 20:24:59.784240 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784210    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58df444989-dw4cd"]
Jul 24 20:24:59.784605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784522    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46970387-be02-4bab-ace6-944b14e17aec" containerName="authorino"
Jul 24 20:24:59.784605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784549    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="46970387-be02-4bab-ace6-944b14e17aec" containerName="authorino"
Jul 24 20:24:59.784605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784562    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="55d415d9-52a7-4947-981d-1ef991a902fe" containerName="authorino"
Jul 24 20:24:59.784605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784567    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="55d415d9-52a7-4947-981d-1ef991a902fe" containerName="authorino"
Jul 24 20:24:59.784605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784578    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="defce69c-7195-4747-8d8c-56ce1d17caf9" containerName="authorino"
Jul 24 20:24:59.784605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784584    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="defce69c-7195-4747-8d8c-56ce1d17caf9" containerName="authorino"
Jul 24 20:24:59.784798 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784636    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="55d415d9-52a7-4947-981d-1ef991a902fe" containerName="authorino"
Jul 24 20:24:59.784798 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784645    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="46970387-be02-4bab-ace6-944b14e17aec" containerName="authorino"
Jul 24 20:24:59.784798 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.784652    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="defce69c-7195-4747-8d8c-56ce1d17caf9" containerName="authorino"
Jul 24 20:24:59.793007 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.792990    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:24:59.796759 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.796733    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 20:24:59.799095 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.799073    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xl8c2\""
Jul 24 20:24:59.799589 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.799217    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 20:24:59.802262 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.802234    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58df444989-dw4cd"]
Jul 24 20:24:59.858303 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.858272    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-427v5\" (UniqueName: \"kubernetes.io/projected/5fdc527b-13b8-442e-a587-72a23dd7a905-kube-api-access-427v5\") pod \"maas-controller-58df444989-dw4cd\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") " pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:24:59.858434 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.858339    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc527b-13b8-442e-a587-72a23dd7a905-webhook-cert\") pod \"maas-controller-58df444989-dw4cd\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") " pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:24:59.959263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.959240    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-427v5\" (UniqueName: \"kubernetes.io/projected/5fdc527b-13b8-442e-a587-72a23dd7a905-kube-api-access-427v5\") pod \"maas-controller-58df444989-dw4cd\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") " pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:24:59.959405 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.959285    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc527b-13b8-442e-a587-72a23dd7a905-webhook-cert\") pod \"maas-controller-58df444989-dw4cd\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") " pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:24:59.961617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.961599    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc527b-13b8-442e-a587-72a23dd7a905-webhook-cert\") pod \"maas-controller-58df444989-dw4cd\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") " pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:24:59.969572 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.969551    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78fd9486f9-868bx"]
Jul 24 20:24:59.971679 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.971659    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-427v5\" (UniqueName: \"kubernetes.io/projected/5fdc527b-13b8-442e-a587-72a23dd7a905-kube-api-access-427v5\") pod \"maas-controller-58df444989-dw4cd\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") " pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:24:59.971927 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.971912    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:24:59.983988 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:24:59.983959    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-868bx"]
Jul 24 20:25:00.060042 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.059963    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-47l5v\" (UniqueName: \"kubernetes.io/projected/4547e654-d567-4b63-bcfe-46c2f50909f3-kube-api-access-47l5v\") pod \"maas-controller-78fd9486f9-868bx\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") " pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:00.060042 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.059993    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4547e654-d567-4b63-bcfe-46c2f50909f3-webhook-cert\") pod \"maas-controller-78fd9486f9-868bx\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") " pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:00.107833 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.107792    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:25:00.160409 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.160381    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-47l5v\" (UniqueName: \"kubernetes.io/projected/4547e654-d567-4b63-bcfe-46c2f50909f3-kube-api-access-47l5v\") pod \"maas-controller-78fd9486f9-868bx\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") " pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:00.160516 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.160417    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4547e654-d567-4b63-bcfe-46c2f50909f3-webhook-cert\") pod \"maas-controller-78fd9486f9-868bx\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") " pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:00.163987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.163938    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4547e654-d567-4b63-bcfe-46c2f50909f3-webhook-cert\") pod \"maas-controller-78fd9486f9-868bx\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") " pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:00.170673 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.170647    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-47l5v\" (UniqueName: \"kubernetes.io/projected/4547e654-d567-4b63-bcfe-46c2f50909f3-kube-api-access-47l5v\") pod \"maas-controller-78fd9486f9-868bx\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") " pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:00.233349 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.233325    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58df444989-dw4cd"]
Jul 24 20:25:00.235606 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:25:00.235570    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fdc527b_13b8_442e_a587_72a23dd7a905.slice/crio-931ca8f1c59cb820e475df81be53056ac744b78f9855e2851f624f41ea6912fe WatchSource:0}: Error finding container 931ca8f1c59cb820e475df81be53056ac744b78f9855e2851f624f41ea6912fe: Status 404 returned error can't find the container with id 931ca8f1c59cb820e475df81be53056ac744b78f9855e2851f624f41ea6912fe
Jul 24 20:25:00.287831 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.287807    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:00.412195 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.412171    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-868bx"]
Jul 24 20:25:00.414008 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:25:00.413981    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4547e654_d567_4b63_bcfe_46c2f50909f3.slice/crio-a93e534a4533fd5b70cea8e5e7e5fa50f86ac1a7487dc3e33b225f4559c0ec90 WatchSource:0}: Error finding container a93e534a4533fd5b70cea8e5e7e5fa50f86ac1a7487dc3e33b225f4559c0ec90: Status 404 returned error can't find the container with id a93e534a4533fd5b70cea8e5e7e5fa50f86ac1a7487dc3e33b225f4559c0ec90
Jul 24 20:25:00.983412 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.983328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58df444989-dw4cd" event={"ID":"5fdc527b-13b8-442e-a587-72a23dd7a905","Type":"ContainerStarted","Data":"931ca8f1c59cb820e475df81be53056ac744b78f9855e2851f624f41ea6912fe"}
Jul 24 20:25:00.984917 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:00.984882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-868bx" event={"ID":"4547e654-d567-4b63-bcfe-46c2f50909f3","Type":"ContainerStarted","Data":"a93e534a4533fd5b70cea8e5e7e5fa50f86ac1a7487dc3e33b225f4559c0ec90"}
Jul 24 20:25:02.995793 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:02.995680    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58df444989-dw4cd" event={"ID":"5fdc527b-13b8-442e-a587-72a23dd7a905","Type":"ContainerStarted","Data":"a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130"}
Jul 24 20:25:02.995793 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:02.995752    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:25:02.997096 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:02.997070    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-868bx" event={"ID":"4547e654-d567-4b63-bcfe-46c2f50909f3","Type":"ContainerStarted","Data":"5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652"}
Jul 24 20:25:02.997225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:02.997124    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:03.017896 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:03.017850    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58df444989-dw4cd" podStartSLOduration=1.529070687 podStartE2EDuration="4.017835436s" totalImagesPullingTime="2.488764749s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:25:03.016885366 +0000 UTC m=+577.943024261" watchObservedRunningTime="2026-07-24 20:25:03.017835436 +0000 UTC m=+577.943974340"
Jul 24 20:25:03.040499 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:03.040452    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78fd9486f9-868bx" podStartSLOduration=1.7203131109999998 podStartE2EDuration="4.040435462s" totalImagesPullingTime="2.320122351s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:25:03.039321956 +0000 UTC m=+577.965460862" watchObservedRunningTime="2026-07-24 20:25:03.040435462 +0000 UTC m=+577.966574398"
Jul 24 20:25:14.004478 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.004445    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:25:14.004988 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.004561    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:25:14.066211 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.066179    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58df444989-dw4cd"]
Jul 24 20:25:14.066398 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.066360    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58df444989-dw4cd" podUID="5fdc527b-13b8-442e-a587-72a23dd7a905" containerName="manager" containerID="cri-o://a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130" gracePeriod=30
Jul 24 20:25:14.306864 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.306835    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:25:14.367476 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.367445    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc527b-13b8-442e-a587-72a23dd7a905-webhook-cert\") pod \"5fdc527b-13b8-442e-a587-72a23dd7a905\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") "
Jul 24 20:25:14.367651 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.367495    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-427v5\" (UniqueName: \"kubernetes.io/projected/5fdc527b-13b8-442e-a587-72a23dd7a905-kube-api-access-427v5\") pod \"5fdc527b-13b8-442e-a587-72a23dd7a905\" (UID: \"5fdc527b-13b8-442e-a587-72a23dd7a905\") "
Jul 24 20:25:14.369682 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.369651    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fdc527b-13b8-442e-a587-72a23dd7a905-kube-api-access-427v5" (OuterVolumeSpecName: "kube-api-access-427v5") pod "5fdc527b-13b8-442e-a587-72a23dd7a905" (UID: "5fdc527b-13b8-442e-a587-72a23dd7a905"). InnerVolumeSpecName "kube-api-access-427v5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:25:14.369682 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.369649    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fdc527b-13b8-442e-a587-72a23dd7a905-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "5fdc527b-13b8-442e-a587-72a23dd7a905" (UID: "5fdc527b-13b8-442e-a587-72a23dd7a905"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:25:14.468903 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.468872    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/5fdc527b-13b8-442e-a587-72a23dd7a905-webhook-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:25:14.468903 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:14.468899    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-427v5\" (UniqueName: \"kubernetes.io/projected/5fdc527b-13b8-442e-a587-72a23dd7a905-kube-api-access-427v5\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:25:15.052034 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.052002    2566 generic.go:356] "Generic (PLEG): container finished" podID="5fdc527b-13b8-442e-a587-72a23dd7a905" containerID="a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130" exitCode=0
Jul 24 20:25:15.052631 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.052044    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58df444989-dw4cd"
Jul 24 20:25:15.052631 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.052086    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58df444989-dw4cd" event={"ID":"5fdc527b-13b8-442e-a587-72a23dd7a905","Type":"ContainerDied","Data":"a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130"}
Jul 24 20:25:15.052631 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.052135    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58df444989-dw4cd" event={"ID":"5fdc527b-13b8-442e-a587-72a23dd7a905","Type":"ContainerDied","Data":"931ca8f1c59cb820e475df81be53056ac744b78f9855e2851f624f41ea6912fe"}
Jul 24 20:25:15.052631 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.052158    2566 scope.go:117] "RemoveContainer" containerID="a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130"
Jul 24 20:25:15.060505 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.060489    2566 scope.go:117] "RemoveContainer" containerID="a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130"
Jul 24 20:25:15.060774 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:25:15.060752    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130\": container with ID starting with a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130 not found: ID does not exist" containerID="a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130"
Jul 24 20:25:15.060837 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.060782    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130"} err="failed to get container status \"a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130\": rpc error: code = NotFound desc = could not find container \"a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130\": container with ID starting with a55b6eb391495014a9b40a7ade64286e851accde51835205653d8de2ad70c130 not found: ID does not exist"
Jul 24 20:25:15.073270 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.073240    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58df444989-dw4cd"]
Jul 24 20:25:15.077052 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.077021    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58df444989-dw4cd"]
Jul 24 20:25:15.692876 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:15.692844    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5fdc527b-13b8-442e-a587-72a23dd7a905" path="/var/lib/kubelet/pods/5fdc527b-13b8-442e-a587-72a23dd7a905/volumes"
Jul 24 20:25:36.168944 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.168909    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"]
Jul 24 20:25:36.169352 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.169243    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5fdc527b-13b8-442e-a587-72a23dd7a905" containerName="manager"
Jul 24 20:25:36.169352 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.169254    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5fdc527b-13b8-442e-a587-72a23dd7a905" containerName="manager"
Jul 24 20:25:36.169352 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.169313    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5fdc527b-13b8-442e-a587-72a23dd7a905" containerName="manager"
Jul 24 20:25:36.174999 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.174979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.177647 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.177623    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Jul 24 20:25:36.177751 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.177624    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-lm72b\""
Jul 24 20:25:36.184222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.184202    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"]
Jul 24 20:25:36.258834 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.258801    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb966c-2f89-4c22-a38a-6fef536e38c9-maas-api-tls\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.259037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.258882    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb966c-2f89-4c22-a38a-6fef536e38c9-openshift-service-ca\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.259037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.258947    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-44zz5\" (UniqueName: \"kubernetes.io/projected/aaeb966c-2f89-4c22-a38a-6fef536e38c9-kube-api-access-44zz5\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.360373 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.360333    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb966c-2f89-4c22-a38a-6fef536e38c9-maas-api-tls\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.360601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.360409    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb966c-2f89-4c22-a38a-6fef536e38c9-openshift-service-ca\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.360601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.360470    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-44zz5\" (UniqueName: \"kubernetes.io/projected/aaeb966c-2f89-4c22-a38a-6fef536e38c9-kube-api-access-44zz5\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.360601 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:25:36.360482    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-serving-cert: secret "maas-api-serving-cert" not found
Jul 24 20:25:36.360601 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:25:36.360585    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/aaeb966c-2f89-4c22-a38a-6fef536e38c9-maas-api-tls podName:aaeb966c-2f89-4c22-a38a-6fef536e38c9 nodeName:}" failed. No retries permitted until 2026-07-24 20:25:36.860563078 +0000 UTC m=+611.786701984 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/aaeb966c-2f89-4c22-a38a-6fef536e38c9-maas-api-tls") pod "maas-api-67d65c95c6-vvtbz" (UID: "aaeb966c-2f89-4c22-a38a-6fef536e38c9") : secret "maas-api-serving-cert" not found
Jul 24 20:25:36.361066 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.361047    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aaeb966c-2f89-4c22-a38a-6fef536e38c9-openshift-service-ca\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.369565 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.369520    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-44zz5\" (UniqueName: \"kubernetes.io/projected/aaeb966c-2f89-4c22-a38a-6fef536e38c9-kube-api-access-44zz5\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.681507 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.681471    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"]
Jul 24 20:25:36.684619 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.684603    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:36.687393 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.687368    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Jul 24 20:25:36.695177 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.695155    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"]
Jul 24 20:25:36.764617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.764575    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bnjc\" (UniqueName: \"kubernetes.io/projected/9a1f2c26-3882-4f04-b1d6-e870df1aa010-kube-api-access-6bnjc\") pod \"payload-pre-processing-687dbb6cf6-hw7pp\" (UID: \"9a1f2c26-3882-4f04-b1d6-e870df1aa010\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:36.764841 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.764661    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1f2c26-3882-4f04-b1d6-e870df1aa010-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hw7pp\" (UID: \"9a1f2c26-3882-4f04-b1d6-e870df1aa010\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:36.865447 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.865407    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1f2c26-3882-4f04-b1d6-e870df1aa010-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hw7pp\" (UID: \"9a1f2c26-3882-4f04-b1d6-e870df1aa010\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:36.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.865572    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bnjc\" (UniqueName: \"kubernetes.io/projected/9a1f2c26-3882-4f04-b1d6-e870df1aa010-kube-api-access-6bnjc\") pod \"payload-pre-processing-687dbb6cf6-hw7pp\" (UID: \"9a1f2c26-3882-4f04-b1d6-e870df1aa010\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:36.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.865603    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb966c-2f89-4c22-a38a-6fef536e38c9-maas-api-tls\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.866275 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.866255    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9a1f2c26-3882-4f04-b1d6-e870df1aa010-plugins-config-volume\") pod \"payload-pre-processing-687dbb6cf6-hw7pp\" (UID: \"9a1f2c26-3882-4f04-b1d6-e870df1aa010\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:36.867886 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.867866    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aaeb966c-2f89-4c22-a38a-6fef536e38c9-maas-api-tls\") pod \"maas-api-67d65c95c6-vvtbz\" (UID: \"aaeb966c-2f89-4c22-a38a-6fef536e38c9\") " pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:36.875262 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.875237    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bnjc\" (UniqueName: \"kubernetes.io/projected/9a1f2c26-3882-4f04-b1d6-e870df1aa010-kube-api-access-6bnjc\") pod \"payload-pre-processing-687dbb6cf6-hw7pp\" (UID: \"9a1f2c26-3882-4f04-b1d6-e870df1aa010\") " pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:36.995935 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:36.995855    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:37.085919 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:37.085885    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:37.133164 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:37.133101    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"]
Jul 24 20:25:37.135828 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:25:37.135790    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9a1f2c26_3882_4f04_b1d6_e870df1aa010.slice/crio-6829c017ba8afbc5c28003cb659349021af9c0a0a17c595fda8738344f30f1ef WatchSource:0}: Error finding container 6829c017ba8afbc5c28003cb659349021af9c0a0a17c595fda8738344f30f1ef: Status 404 returned error can't find the container with id 6829c017ba8afbc5c28003cb659349021af9c0a0a17c595fda8738344f30f1ef
Jul 24 20:25:37.155659 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:37.154880    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp" event={"ID":"9a1f2c26-3882-4f04-b1d6-e870df1aa010","Type":"ContainerStarted","Data":"6829c017ba8afbc5c28003cb659349021af9c0a0a17c595fda8738344f30f1ef"}
Jul 24 20:25:37.233118 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:37.233088    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"]
Jul 24 20:25:37.234591 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:25:37.234564    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaeb966c_2f89_4c22_a38a_6fef536e38c9.slice/crio-e2dd45bd1352920438780a2a9c3b91f59dc17d91b339ab9977f3641abd05b08e WatchSource:0}: Error finding container e2dd45bd1352920438780a2a9c3b91f59dc17d91b339ab9977f3641abd05b08e: Status 404 returned error can't find the container with id e2dd45bd1352920438780a2a9c3b91f59dc17d91b339ab9977f3641abd05b08e
Jul 24 20:25:38.172252 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:38.172161    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz" event={"ID":"aaeb966c-2f89-4c22-a38a-6fef536e38c9","Type":"ContainerStarted","Data":"e2dd45bd1352920438780a2a9c3b91f59dc17d91b339ab9977f3641abd05b08e"}
Jul 24 20:25:41.188358 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:41.188319    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz" event={"ID":"aaeb966c-2f89-4c22-a38a-6fef536e38c9","Type":"ContainerStarted","Data":"084b8b67d070a8b0b0f81138f6e801e791fc1d554db7d8011045264cbbe3f7d9"}
Jul 24 20:25:41.188831 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:41.188503    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:41.189844 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:41.189818    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp" event={"ID":"9a1f2c26-3882-4f04-b1d6-e870df1aa010","Type":"ContainerStarted","Data":"2ff3d0754c45534c297decba412b64ce10fcaa10860e762d7dab9cfe03a9ce0a"}
Jul 24 20:25:41.189947 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:41.189865    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:41.205327 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:41.205286    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz" podStartSLOduration=1.985002489 podStartE2EDuration="5.205273352s" totalImagesPullingTime="3.220270863s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:25:41.204344902 +0000 UTC m=+616.130483807" watchObservedRunningTime="2026-07-24 20:25:41.205273352 +0000 UTC m=+616.131412301"
Jul 24 20:25:41.220574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:41.220515    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp" podStartSLOduration=1.868620095 podStartE2EDuration="5.220501961s" totalImagesPullingTime="3.351881866s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:25:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:25:41.219264317 +0000 UTC m=+616.145403242" watchObservedRunningTime="2026-07-24 20:25:41.220501961 +0000 UTC m=+616.146640865"
Jul 24 20:25:45.212101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:45.212065    2566 generic.go:356] "Generic (PLEG): container finished" podID="3b7372b3-b090-433d-8e7e-f7d6299344ed" containerID="9b0277ed0e756a9a85e932aa2cd96151e8255bd797e479a2e758cb7eb67387ba" exitCode=0
Jul 24 20:25:45.212600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:45.212140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" event={"ID":"3b7372b3-b090-433d-8e7e-f7d6299344ed","Type":"ContainerDied","Data":"9b0277ed0e756a9a85e932aa2cd96151e8255bd797e479a2e758cb7eb67387ba"}
Jul 24 20:25:45.212650 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:45.212595    2566 scope.go:117] "RemoveContainer" containerID="9b0277ed0e756a9a85e932aa2cd96151e8255bd797e479a2e758cb7eb67387ba"
Jul 24 20:25:45.213601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:45.213573    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:25:45.213711 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:45.213655    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:25:45.213772 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:45.213708    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:25:46.218622 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:46.218588    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" event={"ID":"3b7372b3-b090-433d-8e7e-f7d6299344ed","Type":"ContainerStarted","Data":"2722915cdee9494c5109f31fcbba1f119a4406f8cfe0f2441f623cc4f84a703c"}
Jul 24 20:25:46.347586 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:46.347542    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:25:46.347790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:46.347604    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 24 20:25:46.347790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:46.347666    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" podUID="3b7372b3-b090-433d-8e7e-f7d6299344ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 24 20:25:47.201896 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:47.201825    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-67d65c95c6-vvtbz"
Jul 24 20:25:47.353551 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:47.349974    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 24 20:25:47.353551 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:47.350043    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" podUID="3b7372b3-b090-433d-8e7e-f7d6299344ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 24 20:25:48.346966 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:48.346927    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 24 20:25:48.347733 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:48.347263    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" podUID="3b7372b3-b090-433d-8e7e-f7d6299344ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 24 20:25:49.346687 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:49.346653    2566 patch_prober.go:28] interesting pod/maas-default-gateway-openshift-default-78bb78ff78-kpw6c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body=
Jul 24 20:25:49.347679 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:49.347255    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c" podUID="3b7372b3-b090-433d-8e7e-f7d6299344ed" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused"
Jul 24 20:25:50.350689 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:50.350655    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:25:50.351149 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:50.350916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:25:50.351572 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:50.351553    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-78bb78ff78-kpw6c"
Jul 24 20:25:52.198345 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:52.198317    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-687dbb6cf6-hw7pp"
Jul 24 20:25:59.489085 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.489053    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"]
Jul 24 20:25:59.491773 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.491753    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.495221 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.495196    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Jul 24 20:25:59.495355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.495208    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Jul 24 20:25:59.496146 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.496127    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-q65fn\""
Jul 24 20:25:59.496257 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.496166    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Jul 24 20:25:59.503286 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.503265    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"]
Jul 24 20:25:59.581411 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.581371    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.581411 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.581405    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.581643 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.581432    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lszd6\" (UniqueName: \"kubernetes.io/projected/7b2a8d5b-037d-48cc-a5e2-0478119db675-kube-api-access-lszd6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.581643 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.581501    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.581643 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.581568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2a8d5b-037d-48cc-a5e2-0478119db675-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.581643 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.581625    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.682670 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.682634    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lszd6\" (UniqueName: \"kubernetes.io/projected/7b2a8d5b-037d-48cc-a5e2-0478119db675-kube-api-access-lszd6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.682670 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.682675    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.682936 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.682696    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2a8d5b-037d-48cc-a5e2-0478119db675-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.682936 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.682732    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.682936 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.682774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.682936 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.682788    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.683228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.683205    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.683295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.683233    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.683295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.683269    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.684990 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.684968    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b2a8d5b-037d-48cc-a5e2-0478119db675-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.685261 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.685246    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b2a8d5b-037d-48cc-a5e2-0478119db675-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.691091 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.691070    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lszd6\" (UniqueName: \"kubernetes.io/projected/7b2a8d5b-037d-48cc-a5e2-0478119db675-kube-api-access-lszd6\") pod \"e2e-distinct-simulated-kserve-59975d69f9-npztd\" (UID: \"7b2a8d5b-037d-48cc-a5e2-0478119db675\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.803844 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.803737    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:25:59.939243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:25:59.939211    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"]
Jul 24 20:25:59.942995 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:25:59.942967    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b2a8d5b_037d_48cc_a5e2_0478119db675.slice/crio-8e84d34b8644d7cb4413f670ceadea0b79ca24883fa9488a36b1503ad468077a WatchSource:0}: Error finding container 8e84d34b8644d7cb4413f670ceadea0b79ca24883fa9488a36b1503ad468077a: Status 404 returned error can't find the container with id 8e84d34b8644d7cb4413f670ceadea0b79ca24883fa9488a36b1503ad468077a
Jul 24 20:26:00.285960 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:00.285925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd" event={"ID":"7b2a8d5b-037d-48cc-a5e2-0478119db675","Type":"ContainerStarted","Data":"8e84d34b8644d7cb4413f670ceadea0b79ca24883fa9488a36b1503ad468077a"}
Jul 24 20:26:01.298926 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:01.298835    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd" event={"ID":"7b2a8d5b-037d-48cc-a5e2-0478119db675","Type":"ContainerStarted","Data":"d261d67e1746fcea71ceb01b0c3bb81373d7e8e78d90f72e9786572887e883d0"}
Jul 24 20:26:01.320851 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:01.320800    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd" podStartSLOduration=1.240680765 podStartE2EDuration="2.320784254s" totalImagesPullingTime="1.080103489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:25:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:26:01.319116462 +0000 UTC m=+636.245255370" watchObservedRunningTime="2026-07-24 20:26:01.320784254 +0000 UTC m=+636.246923160"
Jul 24 20:26:09.804573 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:09.804523    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:26:09.804980 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:09.804606    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:26:09.817082 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:09.817057    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:26:10.349022 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:10.348986    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-npztd"
Jul 24 20:26:39.602750 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.602712    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"]
Jul 24 20:26:39.607370 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.607347    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.611373 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.611353    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Jul 24 20:26:39.622601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.622574    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"]
Jul 24 20:26:39.642806 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.642773    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.643101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.643076    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.643268 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.643251    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e7a622-87da-438c-880b-a3a6fe34df2c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.643443 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.643418    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.643555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.643452    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.643555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.643486    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpx5g\" (UniqueName: \"kubernetes.io/projected/d5e7a622-87da-438c-880b-a3a6fe34df2c-kube-api-access-hpx5g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.743952 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.743919    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.743952 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.743957    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.744212 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.743986    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpx5g\" (UniqueName: \"kubernetes.io/projected/d5e7a622-87da-438c-880b-a3a6fe34df2c-kube-api-access-hpx5g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.744212 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.744046    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.744212 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.744097    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.744212 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.744134    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e7a622-87da-438c-880b-a3a6fe34df2c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.744421 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.744403    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.744597 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.744574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-home\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.744783 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.744754    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-model-cache\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.746578 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.746552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d5e7a622-87da-438c-880b-a3a6fe34df2c-dshm\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.747004 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.746988    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d5e7a622-87da-438c-880b-a3a6fe34df2c-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.753248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.753216    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpx5g\" (UniqueName: \"kubernetes.io/projected/d5e7a622-87da-438c-880b-a3a6fe34df2c-kube-api-access-hpx5g\") pod \"facebook-opt-125m-simulated-kserve-74b68447df-ptcfh\" (UID: \"d5e7a622-87da-438c-880b-a3a6fe34df2c\") " pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:39.918174 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:39.918074    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:40.056471 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:40.056444    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"]
Jul 24 20:26:40.057821 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:26:40.057791    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5e7a622_87da_438c_880b_a3a6fe34df2c.slice/crio-12aeba6690553da9710365e57aa72ce70242a841f151f3698f7ccad7f1bca84e WatchSource:0}: Error finding container 12aeba6690553da9710365e57aa72ce70242a841f151f3698f7ccad7f1bca84e: Status 404 returned error can't find the container with id 12aeba6690553da9710365e57aa72ce70242a841f151f3698f7ccad7f1bca84e
Jul 24 20:26:40.059267 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:40.059245    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 20:26:40.472362 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:40.472330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh" event={"ID":"d5e7a622-87da-438c-880b-a3a6fe34df2c","Type":"ContainerStarted","Data":"c3b0f21bb73dc5be43f6abfb9be8210b3280a0179f1b693599f92bf84e0f0a8c"}
Jul 24 20:26:40.472570 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:40.472378    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh" event={"ID":"d5e7a622-87da-438c-880b-a3a6fe34df2c","Type":"ContainerStarted","Data":"12aeba6690553da9710365e57aa72ce70242a841f151f3698f7ccad7f1bca84e"}
Jul 24 20:26:40.491194 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:40.491143    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh" podStartSLOduration=1.167249985 podStartE2EDuration="1.491125092s" totalImagesPullingTime="323.875107ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:26:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:26:40.490047988 +0000 UTC m=+675.416186913" watchObservedRunningTime="2026-07-24 20:26:40.491125092 +0000 UTC m=+675.417263994"
Jul 24 20:26:49.918454 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:49.918410    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:49.919055 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:49.918486    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:49.931220 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:49.931190    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:26:50.528987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:26:50.528954    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-74b68447df-ptcfh"
Jul 24 20:27:31.378174 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.378085    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-95c9487bd-96jhh"]
Jul 24 20:27:31.381592 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.381568    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.393071 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.393031    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-95c9487bd-96jhh"]
Jul 24 20:27:31.460559 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.460512    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c32077a-9e9c-4af4-a4ea-70a1adf44296-tls-cert\") pod \"authorino-95c9487bd-96jhh\" (UID: \"7c32077a-9e9c-4af4-a4ea-70a1adf44296\") " pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.460668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.460578    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqx8q\" (UniqueName: \"kubernetes.io/projected/7c32077a-9e9c-4af4-a4ea-70a1adf44296-kube-api-access-fqx8q\") pod \"authorino-95c9487bd-96jhh\" (UID: \"7c32077a-9e9c-4af4-a4ea-70a1adf44296\") " pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.561797 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.561765    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c32077a-9e9c-4af4-a4ea-70a1adf44296-tls-cert\") pod \"authorino-95c9487bd-96jhh\" (UID: \"7c32077a-9e9c-4af4-a4ea-70a1adf44296\") " pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.561910 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.561821    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqx8q\" (UniqueName: \"kubernetes.io/projected/7c32077a-9e9c-4af4-a4ea-70a1adf44296-kube-api-access-fqx8q\") pod \"authorino-95c9487bd-96jhh\" (UID: \"7c32077a-9e9c-4af4-a4ea-70a1adf44296\") " pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.564042 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.564022    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/7c32077a-9e9c-4af4-a4ea-70a1adf44296-tls-cert\") pod \"authorino-95c9487bd-96jhh\" (UID: \"7c32077a-9e9c-4af4-a4ea-70a1adf44296\") " pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.570941 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.570918    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqx8q\" (UniqueName: \"kubernetes.io/projected/7c32077a-9e9c-4af4-a4ea-70a1adf44296-kube-api-access-fqx8q\") pod \"authorino-95c9487bd-96jhh\" (UID: \"7c32077a-9e9c-4af4-a4ea-70a1adf44296\") " pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.690959 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.690895    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-95c9487bd-96jhh"
Jul 24 20:27:31.815499 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:31.815474    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-95c9487bd-96jhh"]
Jul 24 20:27:31.817339 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:27:31.817311    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7c32077a_9e9c_4af4_a4ea_70a1adf44296.slice/crio-019a2874637042ce205b21fcbaeaff317a759961d04b876698b868a566eb0a9e WatchSource:0}: Error finding container 019a2874637042ce205b21fcbaeaff317a759961d04b876698b868a566eb0a9e: Status 404 returned error can't find the container with id 019a2874637042ce205b21fcbaeaff317a759961d04b876698b868a566eb0a9e
Jul 24 20:27:32.713330 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:32.713289    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-95c9487bd-96jhh" event={"ID":"7c32077a-9e9c-4af4-a4ea-70a1adf44296","Type":"ContainerStarted","Data":"3ce3cf888a8e9a071166a67b2a334800576048db0c8091ef07907f6122f2c088"}
Jul 24 20:27:32.713330 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:32.713338    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-95c9487bd-96jhh" event={"ID":"7c32077a-9e9c-4af4-a4ea-70a1adf44296","Type":"ContainerStarted","Data":"019a2874637042ce205b21fcbaeaff317a759961d04b876698b868a566eb0a9e"}
Jul 24 20:27:32.730862 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:32.730801    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-95c9487bd-96jhh" podStartSLOduration=1.316723429 podStartE2EDuration="1.730786653s" totalImagesPullingTime="414.063224ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:27:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:27:32.728869767 +0000 UTC m=+727.655008669" watchObservedRunningTime="2026-07-24 20:27:32.730786653 +0000 UTC m=+727.656925557"
Jul 24 20:27:32.758285 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:32.758243    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59f6fcf49c-x6brm"]
Jul 24 20:27:32.758454 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:32.758426    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59f6fcf49c-x6brm" podUID="baeb603a-ff5c-4996-8456-f39138c5a550" containerName="authorino" containerID="cri-o://5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353" gracePeriod=30
Jul 24 20:27:33.007461 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.007436    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:27:33.076722 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.076695    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baeb603a-ff5c-4996-8456-f39138c5a550-tls-cert\") pod \"baeb603a-ff5c-4996-8456-f39138c5a550\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") "
Jul 24 20:27:33.076873 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.076821    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jf9q7\" (UniqueName: \"kubernetes.io/projected/baeb603a-ff5c-4996-8456-f39138c5a550-kube-api-access-jf9q7\") pod \"baeb603a-ff5c-4996-8456-f39138c5a550\" (UID: \"baeb603a-ff5c-4996-8456-f39138c5a550\") "
Jul 24 20:27:33.078787 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.078734    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baeb603a-ff5c-4996-8456-f39138c5a550-kube-api-access-jf9q7" (OuterVolumeSpecName: "kube-api-access-jf9q7") pod "baeb603a-ff5c-4996-8456-f39138c5a550" (UID: "baeb603a-ff5c-4996-8456-f39138c5a550"). InnerVolumeSpecName "kube-api-access-jf9q7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:27:33.086954 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.086925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/baeb603a-ff5c-4996-8456-f39138c5a550-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "baeb603a-ff5c-4996-8456-f39138c5a550" (UID: "baeb603a-ff5c-4996-8456-f39138c5a550"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:27:33.178222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.178188    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jf9q7\" (UniqueName: \"kubernetes.io/projected/baeb603a-ff5c-4996-8456-f39138c5a550-kube-api-access-jf9q7\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:27:33.178222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.178216    2566 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/baeb603a-ff5c-4996-8456-f39138c5a550-tls-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:27:33.720505 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.720471    2566 generic.go:356] "Generic (PLEG): container finished" podID="baeb603a-ff5c-4996-8456-f39138c5a550" containerID="5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353" exitCode=0
Jul 24 20:27:33.720903 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.720540    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59f6fcf49c-x6brm"
Jul 24 20:27:33.720903 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.720562    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f6fcf49c-x6brm" event={"ID":"baeb603a-ff5c-4996-8456-f39138c5a550","Type":"ContainerDied","Data":"5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353"}
Jul 24 20:27:33.720903 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.720602    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59f6fcf49c-x6brm" event={"ID":"baeb603a-ff5c-4996-8456-f39138c5a550","Type":"ContainerDied","Data":"10138c61504a86d88649daf2ac26760a6c01d7303f0418db348c2b662b0ba10c"}
Jul 24 20:27:33.720903 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.720624    2566 scope.go:117] "RemoveContainer" containerID="5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353"
Jul 24 20:27:33.728998 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.728951    2566 scope.go:117] "RemoveContainer" containerID="5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353"
Jul 24 20:27:33.729261 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:27:33.729242    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353\": container with ID starting with 5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353 not found: ID does not exist" containerID="5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353"
Jul 24 20:27:33.729337 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.729268    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353"} err="failed to get container status \"5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353\": rpc error: code = NotFound desc = could not find container \"5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353\": container with ID starting with 5a47115a2dd1656adae6e212018fee504ecfed20909290af12e1f18a0364b353 not found: ID does not exist"
Jul 24 20:27:33.738650 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.738629    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59f6fcf49c-x6brm"]
Jul 24 20:27:33.742470 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:33.742447    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59f6fcf49c-x6brm"]
Jul 24 20:27:35.692392 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:27:35.692357    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baeb603a-ff5c-4996-8456-f39138c5a550" path="/var/lib/kubelet/pods/baeb603a-ff5c-4996-8456-f39138c5a550/volumes"
Jul 24 20:28:20.578101 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.578068    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-868bx"]
Jul 24 20:28:20.578582 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.578278    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78fd9486f9-868bx" podUID="4547e654-d567-4b63-bcfe-46c2f50909f3" containerName="manager" containerID="cri-o://5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652" gracePeriod=30
Jul 24 20:28:20.869597 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.869571    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:28:20.881317 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.881291    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4547e654-d567-4b63-bcfe-46c2f50909f3-webhook-cert\") pod \"4547e654-d567-4b63-bcfe-46c2f50909f3\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") "
Jul 24 20:28:20.881446 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.881346    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-47l5v\" (UniqueName: \"kubernetes.io/projected/4547e654-d567-4b63-bcfe-46c2f50909f3-kube-api-access-47l5v\") pod \"4547e654-d567-4b63-bcfe-46c2f50909f3\" (UID: \"4547e654-d567-4b63-bcfe-46c2f50909f3\") "
Jul 24 20:28:20.883425 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.883402    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4547e654-d567-4b63-bcfe-46c2f50909f3-kube-api-access-47l5v" (OuterVolumeSpecName: "kube-api-access-47l5v") pod "4547e654-d567-4b63-bcfe-46c2f50909f3" (UID: "4547e654-d567-4b63-bcfe-46c2f50909f3"). InnerVolumeSpecName "kube-api-access-47l5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:28:20.883685 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.883667    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4547e654-d567-4b63-bcfe-46c2f50909f3-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "4547e654-d567-4b63-bcfe-46c2f50909f3" (UID: "4547e654-d567-4b63-bcfe-46c2f50909f3"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:28:20.936414 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.936380    2566 generic.go:356] "Generic (PLEG): container finished" podID="4547e654-d567-4b63-bcfe-46c2f50909f3" containerID="5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652" exitCode=0
Jul 24 20:28:20.936623 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.936426    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-868bx"
Jul 24 20:28:20.936623 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.936464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-868bx" event={"ID":"4547e654-d567-4b63-bcfe-46c2f50909f3","Type":"ContainerDied","Data":"5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652"}
Jul 24 20:28:20.936623 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.936553    2566 scope.go:117] "RemoveContainer" containerID="5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652"
Jul 24 20:28:20.936623 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.936578    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-868bx" event={"ID":"4547e654-d567-4b63-bcfe-46c2f50909f3","Type":"ContainerDied","Data":"a93e534a4533fd5b70cea8e5e7e5fa50f86ac1a7487dc3e33b225f4559c0ec90"}
Jul 24 20:28:20.945668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.945633    2566 scope.go:117] "RemoveContainer" containerID="5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652"
Jul 24 20:28:20.945887 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:28:20.945867    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652\": container with ID starting with 5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652 not found: ID does not exist" containerID="5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652"
Jul 24 20:28:20.945939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.945895    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652"} err="failed to get container status \"5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652\": rpc error: code = NotFound desc = could not find container \"5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652\": container with ID starting with 5d665d0f6ac152270ff1cd965880dc8fc9d13482a99b596b7e911d837dd55652 not found: ID does not exist"
Jul 24 20:28:20.958375 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.958352    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-868bx"]
Jul 24 20:28:20.964284 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.964263    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-868bx"]
Jul 24 20:28:20.982082 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.982063    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/4547e654-d567-4b63-bcfe-46c2f50909f3-webhook-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:28:20.982153 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:20.982085    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-47l5v\" (UniqueName: \"kubernetes.io/projected/4547e654-d567-4b63-bcfe-46c2f50909f3-kube-api-access-47l5v\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:28:21.691753 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:28:21.691716    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4547e654-d567-4b63-bcfe-46c2f50909f3" path="/var/lib/kubelet/pods/4547e654-d567-4b63-bcfe-46c2f50909f3/volumes"
Jul 24 20:29:21.815496 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.815457    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78fd9486f9-9mghj"]
Jul 24 20:29:21.816034 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.815973    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baeb603a-ff5c-4996-8456-f39138c5a550" containerName="authorino"
Jul 24 20:29:21.816034 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.815995    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="baeb603a-ff5c-4996-8456-f39138c5a550" containerName="authorino"
Jul 24 20:29:21.816034 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.816007    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4547e654-d567-4b63-bcfe-46c2f50909f3" containerName="manager"
Jul 24 20:29:21.816034 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.816017    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="4547e654-d567-4b63-bcfe-46c2f50909f3" containerName="manager"
Jul 24 20:29:21.816245 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.816113    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="baeb603a-ff5c-4996-8456-f39138c5a550" containerName="authorino"
Jul 24 20:29:21.816245 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.816127    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="4547e654-d567-4b63-bcfe-46c2f50909f3" containerName="manager"
Jul 24 20:29:21.818952 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.818931    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:21.822489 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.822464    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xl8c2\""
Jul 24 20:29:21.822596 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.822499    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 20:29:21.822596 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.822504    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 20:29:21.827168 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.827144    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-9mghj"]
Jul 24 20:29:21.869447 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.869420    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f39a9090-0e59-48b9-a7c8-999572f6a767-webhook-cert\") pod \"maas-controller-78fd9486f9-9mghj\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") " pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:21.869564 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.869458    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rgxf2\" (UniqueName: \"kubernetes.io/projected/f39a9090-0e59-48b9-a7c8-999572f6a767-kube-api-access-rgxf2\") pod \"maas-controller-78fd9486f9-9mghj\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") " pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:21.970220 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.970185    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rgxf2\" (UniqueName: \"kubernetes.io/projected/f39a9090-0e59-48b9-a7c8-999572f6a767-kube-api-access-rgxf2\") pod \"maas-controller-78fd9486f9-9mghj\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") " pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:21.970348 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.970293    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f39a9090-0e59-48b9-a7c8-999572f6a767-webhook-cert\") pod \"maas-controller-78fd9486f9-9mghj\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") " pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:21.972424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.972405    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f39a9090-0e59-48b9-a7c8-999572f6a767-webhook-cert\") pod \"maas-controller-78fd9486f9-9mghj\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") " pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:21.978180 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:21.978156    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rgxf2\" (UniqueName: \"kubernetes.io/projected/f39a9090-0e59-48b9-a7c8-999572f6a767-kube-api-access-rgxf2\") pod \"maas-controller-78fd9486f9-9mghj\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") " pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:22.130350 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:22.130323    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:22.266653 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:22.266618    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-9mghj"]
Jul 24 20:29:22.269757 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:29:22.269723    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf39a9090_0e59_48b9_a7c8_999572f6a767.slice/crio-4076461bff9ccc2568c275eba20fa8caa99d3d610188c747ba7c7ccdc50dfab3 WatchSource:0}: Error finding container 4076461bff9ccc2568c275eba20fa8caa99d3d610188c747ba7c7ccdc50dfab3: Status 404 returned error can't find the container with id 4076461bff9ccc2568c275eba20fa8caa99d3d610188c747ba7c7ccdc50dfab3
Jul 24 20:29:23.224309 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:23.224274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-9mghj" event={"ID":"f39a9090-0e59-48b9-a7c8-999572f6a767","Type":"ContainerStarted","Data":"80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6"}
Jul 24 20:29:23.224733 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:23.224330    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-9mghj" event={"ID":"f39a9090-0e59-48b9-a7c8-999572f6a767","Type":"ContainerStarted","Data":"4076461bff9ccc2568c275eba20fa8caa99d3d610188c747ba7c7ccdc50dfab3"}
Jul 24 20:29:23.224733 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:23.224358    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:23.242953 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:23.242908    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78fd9486f9-9mghj" podStartSLOduration=1.7242812349999999 podStartE2EDuration="2.242891891s" totalImagesPullingTime="518.610656ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:29:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:29:23.240782692 +0000 UTC m=+838.166921597" watchObservedRunningTime="2026-07-24 20:29:23.242891891 +0000 UTC m=+838.169030795"
Jul 24 20:29:34.231652 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:34.231613    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:38.927540 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:38.927498    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-9mghj"]
Jul 24 20:29:38.928154 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:38.927707    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-78fd9486f9-9mghj" podUID="f39a9090-0e59-48b9-a7c8-999572f6a767" containerName="manager" containerID="cri-o://80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6" gracePeriod=30
Jul 24 20:29:39.184023 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.183957    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:39.300807 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.300774    2566 generic.go:356] "Generic (PLEG): container finished" podID="f39a9090-0e59-48b9-a7c8-999572f6a767" containerID="80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6" exitCode=0
Jul 24 20:29:39.300985 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.300830    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-9mghj"
Jul 24 20:29:39.300985 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.300833    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-9mghj" event={"ID":"f39a9090-0e59-48b9-a7c8-999572f6a767","Type":"ContainerDied","Data":"80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6"}
Jul 24 20:29:39.300985 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.300968    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-9mghj" event={"ID":"f39a9090-0e59-48b9-a7c8-999572f6a767","Type":"ContainerDied","Data":"4076461bff9ccc2568c275eba20fa8caa99d3d610188c747ba7c7ccdc50dfab3"}
Jul 24 20:29:39.301135 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.300992    2566 scope.go:117] "RemoveContainer" containerID="80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6"
Jul 24 20:29:39.309181 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.309155    2566 scope.go:117] "RemoveContainer" containerID="80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6"
Jul 24 20:29:39.309508 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:29:39.309480    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6\": container with ID starting with 80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6 not found: ID does not exist" containerID="80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6"
Jul 24 20:29:39.309725 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.309517    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6"} err="failed to get container status \"80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6\": rpc error: code = NotFound desc = could not find container \"80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6\": container with ID starting with 80b40a37358c1f266dcf3f95e1a54305f4009afbf2e3267813e24b71fd43b8c6 not found: ID does not exist"
Jul 24 20:29:39.316832 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.316811    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f39a9090-0e59-48b9-a7c8-999572f6a767-webhook-cert\") pod \"f39a9090-0e59-48b9-a7c8-999572f6a767\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") "
Jul 24 20:29:39.316924 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.316887    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rgxf2\" (UniqueName: \"kubernetes.io/projected/f39a9090-0e59-48b9-a7c8-999572f6a767-kube-api-access-rgxf2\") pod \"f39a9090-0e59-48b9-a7c8-999572f6a767\" (UID: \"f39a9090-0e59-48b9-a7c8-999572f6a767\") "
Jul 24 20:29:39.318770 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.318745    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f39a9090-0e59-48b9-a7c8-999572f6a767-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "f39a9090-0e59-48b9-a7c8-999572f6a767" (UID: "f39a9090-0e59-48b9-a7c8-999572f6a767"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:29:39.318855 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.318780    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f39a9090-0e59-48b9-a7c8-999572f6a767-kube-api-access-rgxf2" (OuterVolumeSpecName: "kube-api-access-rgxf2") pod "f39a9090-0e59-48b9-a7c8-999572f6a767" (UID: "f39a9090-0e59-48b9-a7c8-999572f6a767"). InnerVolumeSpecName "kube-api-access-rgxf2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:29:39.417958 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.417923    2566 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f39a9090-0e59-48b9-a7c8-999572f6a767-webhook-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:29:39.417958 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.417953    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rgxf2\" (UniqueName: \"kubernetes.io/projected/f39a9090-0e59-48b9-a7c8-999572f6a767-kube-api-access-rgxf2\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:29:39.623340 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.623313    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-9mghj"]
Jul 24 20:29:39.626434 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.626408    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-9mghj"]
Jul 24 20:29:39.692307 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:39.692274    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f39a9090-0e59-48b9-a7c8-999572f6a767" path="/var/lib/kubelet/pods/f39a9090-0e59-48b9-a7c8-999572f6a767/volumes"
Jul 24 20:29:40.460412 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.460383    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-78fd9486f9-flgh8"]
Jul 24 20:29:40.460835 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.460791    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f39a9090-0e59-48b9-a7c8-999572f6a767" containerName="manager"
Jul 24 20:29:40.460835 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.460806    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f39a9090-0e59-48b9-a7c8-999572f6a767" containerName="manager"
Jul 24 20:29:40.460939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.460864    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f39a9090-0e59-48b9-a7c8-999572f6a767" containerName="manager"
Jul 24 20:29:40.465340 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.465320    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.468059 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.468036    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Jul 24 20:29:40.469218 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.469201    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-xl8c2\""
Jul 24 20:29:40.469323 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.469201    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Jul 24 20:29:40.472404 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.472381    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-flgh8"]
Jul 24 20:29:40.528229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.528196    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef6d78b9-f7e7-4909-ac84-df443ef0ace1-webhook-cert\") pod \"maas-controller-78fd9486f9-flgh8\" (UID: \"ef6d78b9-f7e7-4909-ac84-df443ef0ace1\") " pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.528229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.528226    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlddp\" (UniqueName: \"kubernetes.io/projected/ef6d78b9-f7e7-4909-ac84-df443ef0ace1-kube-api-access-dlddp\") pod \"maas-controller-78fd9486f9-flgh8\" (UID: \"ef6d78b9-f7e7-4909-ac84-df443ef0ace1\") " pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.628882 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.628842    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef6d78b9-f7e7-4909-ac84-df443ef0ace1-webhook-cert\") pod \"maas-controller-78fd9486f9-flgh8\" (UID: \"ef6d78b9-f7e7-4909-ac84-df443ef0ace1\") " pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.628882 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.628882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlddp\" (UniqueName: \"kubernetes.io/projected/ef6d78b9-f7e7-4909-ac84-df443ef0ace1-kube-api-access-dlddp\") pod \"maas-controller-78fd9486f9-flgh8\" (UID: \"ef6d78b9-f7e7-4909-ac84-df443ef0ace1\") " pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.631258 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.631238    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ef6d78b9-f7e7-4909-ac84-df443ef0ace1-webhook-cert\") pod \"maas-controller-78fd9486f9-flgh8\" (UID: \"ef6d78b9-f7e7-4909-ac84-df443ef0ace1\") " pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.637834 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.637808    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlddp\" (UniqueName: \"kubernetes.io/projected/ef6d78b9-f7e7-4909-ac84-df443ef0ace1-kube-api-access-dlddp\") pod \"maas-controller-78fd9486f9-flgh8\" (UID: \"ef6d78b9-f7e7-4909-ac84-df443ef0ace1\") " pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.776780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.776689    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:40.915544 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:40.915436    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-78fd9486f9-flgh8"]
Jul 24 20:29:40.917842 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:29:40.917810    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef6d78b9_f7e7_4909_ac84_df443ef0ace1.slice/crio-5ccabbc04714048cc0bc344f7464a9a7ca994582c43b6a7bd2cdbca9e86b469c WatchSource:0}: Error finding container 5ccabbc04714048cc0bc344f7464a9a7ca994582c43b6a7bd2cdbca9e86b469c: Status 404 returned error can't find the container with id 5ccabbc04714048cc0bc344f7464a9a7ca994582c43b6a7bd2cdbca9e86b469c
Jul 24 20:29:41.312957 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:41.312917    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-flgh8" event={"ID":"ef6d78b9-f7e7-4909-ac84-df443ef0ace1","Type":"ContainerStarted","Data":"2d01458dc4c9b9ce30c7d24342d38308549818d9d28f551d89d3b3a74cf0456f"}
Jul 24 20:29:41.313128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:41.312989    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-78fd9486f9-flgh8" event={"ID":"ef6d78b9-f7e7-4909-ac84-df443ef0ace1","Type":"ContainerStarted","Data":"5ccabbc04714048cc0bc344f7464a9a7ca994582c43b6a7bd2cdbca9e86b469c"}
Jul 24 20:29:41.313128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:41.313019    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:29:41.333093 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:41.333035    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-78fd9486f9-flgh8" podStartSLOduration=1.009552374 podStartE2EDuration="1.333018326s" totalImagesPullingTime="323.465952ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:29:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:29:41.331733292 +0000 UTC m=+856.257872230" watchObservedRunningTime="2026-07-24 20:29:41.333018326 +0000 UTC m=+856.259157231"
Jul 24 20:29:52.319973 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:29:52.319945    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-78fd9486f9-flgh8"
Jul 24 20:30:00.135265 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:00.135229    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp"]
Jul 24 20:30:00.139315 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:00.139287    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp"
Jul 24 20:30:00.142163 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:00.142139    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 20:30:00.146090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:00.146065    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp"]
Jul 24 20:30:00.155059 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:00.154906    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp"
Jul 24 20:30:00.297582 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:00.297550    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp"]
Jul 24 20:30:00.299983 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:30:00.299948    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc35b0bec_4d05_4a6b_a563_f23ecc468d4b.slice/crio-4e1dad1599e74084673667fa3e9e9be472a2e726572c32eb0cbf6d8133f6070f WatchSource:0}: Error finding container 4e1dad1599e74084673667fa3e9e9be472a2e726572c32eb0cbf6d8133f6070f: Status 404 returned error can't find the container with id 4e1dad1599e74084673667fa3e9e9be472a2e726572c32eb0cbf6d8133f6070f
Jul 24 20:30:00.403124 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:00.403042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp" event={"ID":"c35b0bec-4d05-4a6b-a563-f23ecc468d4b","Type":"ContainerStarted","Data":"4e1dad1599e74084673667fa3e9e9be472a2e726572c32eb0cbf6d8133f6070f"}
Jul 24 20:30:06.433415 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:06.433352    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp" event={"ID":"c35b0bec-4d05-4a6b-a563-f23ecc468d4b","Type":"ContainerStarted","Data":"3ca34e57f0eddbc585db0aa7742754be30853b4531102b241c78fca67527f6ae"}
Jul 24 20:30:06.450252 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:06.450204    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp" podStartSLOduration=0.962773835 podStartE2EDuration="6.450189578s" totalImagesPullingTime="5.487415743s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:30:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:30:06.448385079 +0000 UTC m=+881.374524008" watchObservedRunningTime="2026-07-24 20:30:06.450189578 +0000 UTC m=+881.376328507"
Jul 24 20:30:07.440268 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:07.440226    2566 generic.go:356] "Generic (PLEG): container finished" podID="c35b0bec-4d05-4a6b-a563-f23ecc468d4b" containerID="3ca34e57f0eddbc585db0aa7742754be30853b4531102b241c78fca67527f6ae" exitCode=0
Jul 24 20:30:07.440669 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:07.440276    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp" event={"ID":"c35b0bec-4d05-4a6b-a563-f23ecc468d4b","Type":"ContainerDied","Data":"3ca34e57f0eddbc585db0aa7742754be30853b4531102b241c78fca67527f6ae"}
Jul 24 20:30:08.579333 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:08.579310    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp"
Jul 24 20:30:09.450723 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:09.450685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp"
Jul 24 20:30:09.450891 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:09.450688    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748750-k96vp" event={"ID":"c35b0bec-4d05-4a6b-a563-f23ecc468d4b","Type":"ContainerDied","Data":"4e1dad1599e74084673667fa3e9e9be472a2e726572c32eb0cbf6d8133f6070f"}
Jul 24 20:30:09.450891 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:30:09.450824    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e1dad1599e74084673667fa3e9e9be472a2e726572c32eb0cbf6d8133f6070f"
Jul 24 20:34:24.241539 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:24.241499    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"]
Jul 24 20:34:24.242059 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:24.241738    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb" podUID="1915e104-0bb9-43cb-a7fa-b4b0a82586ce" containerName="manager" containerID="cri-o://b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f" gracePeriod=10
Jul 24 20:34:26.593045 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.593024    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:34:26.627458 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.627433    2566 generic.go:356] "Generic (PLEG): container finished" podID="1915e104-0bb9-43cb-a7fa-b4b0a82586ce" containerID="b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f" exitCode=0
Jul 24 20:34:26.627593 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.627474    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"
Jul 24 20:34:26.627593 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.627520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb" event={"ID":"1915e104-0bb9-43cb-a7fa-b4b0a82586ce","Type":"ContainerDied","Data":"b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f"}
Jul 24 20:34:26.627682 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.627593    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb" event={"ID":"1915e104-0bb9-43cb-a7fa-b4b0a82586ce","Type":"ContainerDied","Data":"a730a48f699db787ef4eaaa8243ca5611b2d8df43cc74d061e0d9787f1d769a0"}
Jul 24 20:34:26.627682 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.627625    2566 scope.go:117] "RemoveContainer" containerID="b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f"
Jul 24 20:34:26.638853 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.638836    2566 scope.go:117] "RemoveContainer" containerID="b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f"
Jul 24 20:34:26.639103 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:34:26.639085    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f\": container with ID starting with b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f not found: ID does not exist" containerID="b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f"
Jul 24 20:34:26.639186 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.639112    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f"} err="failed to get container status \"b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f\": rpc error: code = NotFound desc = could not find container \"b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f\": container with ID starting with b4eb99580fa870c0a9ba900e1d5167d8761e3e4afe8dfb8799702b7f67563c5f not found: ID does not exist"
Jul 24 20:34:26.682424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.682405    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-extensions-socket-volume\") pod \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") "
Jul 24 20:34:26.682519 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.682463    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpvz5\" (UniqueName: \"kubernetes.io/projected/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-kube-api-access-jpvz5\") pod \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\" (UID: \"1915e104-0bb9-43cb-a7fa-b4b0a82586ce\") "
Jul 24 20:34:26.682759 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.682736    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "1915e104-0bb9-43cb-a7fa-b4b0a82586ce" (UID: "1915e104-0bb9-43cb-a7fa-b4b0a82586ce"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:34:26.684222 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.684197    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-kube-api-access-jpvz5" (OuterVolumeSpecName: "kube-api-access-jpvz5") pod "1915e104-0bb9-43cb-a7fa-b4b0a82586ce" (UID: "1915e104-0bb9-43cb-a7fa-b4b0a82586ce"). InnerVolumeSpecName "kube-api-access-jpvz5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:34:26.783842 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.783790    2566 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-extensions-socket-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:34:26.783842 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.783812    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvz5\" (UniqueName: \"kubernetes.io/projected/1915e104-0bb9-43cb-a7fa-b4b0a82586ce-kube-api-access-jpvz5\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:34:26.949299 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.949276    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"]
Jul 24 20:34:26.952942 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:26.952923    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-tcprb"]
Jul 24 20:34:27.691707 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:34:27.691670    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1915e104-0bb9-43cb-a7fa-b4b0a82586ce" path="/var/lib/kubelet/pods/1915e104-0bb9-43cb-a7fa-b4b0a82586ce/volumes"
Jul 24 20:35:25.372330 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.372292    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"]
Jul 24 20:35:25.372941 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.372810    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c35b0bec-4d05-4a6b-a563-f23ecc468d4b" containerName="cleanup"
Jul 24 20:35:25.372941 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.372829    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="c35b0bec-4d05-4a6b-a563-f23ecc468d4b" containerName="cleanup"
Jul 24 20:35:25.372941 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.372857    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1915e104-0bb9-43cb-a7fa-b4b0a82586ce" containerName="manager"
Jul 24 20:35:25.372941 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.372866    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1915e104-0bb9-43cb-a7fa-b4b0a82586ce" containerName="manager"
Jul 24 20:35:25.373161 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.372953    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1915e104-0bb9-43cb-a7fa-b4b0a82586ce" containerName="manager"
Jul 24 20:35:25.373161 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.372965    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="c35b0bec-4d05-4a6b-a563-f23ecc468d4b" containerName="cleanup"
Jul 24 20:35:25.375116 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.375096    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.377824 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.377806    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wnd75\""
Jul 24 20:35:25.386585 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.386564    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"]
Jul 24 20:35:25.450787 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.450746    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tfg9\" (UniqueName: \"kubernetes.io/projected/00c92151-a9bb-4bae-9040-aaf58be425f0-kube-api-access-6tfg9\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4wc\" (UID: \"00c92151-a9bb-4bae-9040-aaf58be425f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.450907 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.450835    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00c92151-a9bb-4bae-9040-aaf58be425f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4wc\" (UID: \"00c92151-a9bb-4bae-9040-aaf58be425f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.552073 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.552033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tfg9\" (UniqueName: \"kubernetes.io/projected/00c92151-a9bb-4bae-9040-aaf58be425f0-kube-api-access-6tfg9\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4wc\" (UID: \"00c92151-a9bb-4bae-9040-aaf58be425f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.552220 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.552100    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00c92151-a9bb-4bae-9040-aaf58be425f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4wc\" (UID: \"00c92151-a9bb-4bae-9040-aaf58be425f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.552410 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.552395    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/00c92151-a9bb-4bae-9040-aaf58be425f0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4wc\" (UID: \"00c92151-a9bb-4bae-9040-aaf58be425f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.559944 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.559925    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tfg9\" (UniqueName: \"kubernetes.io/projected/00c92151-a9bb-4bae-9040-aaf58be425f0-kube-api-access-6tfg9\") pod \"kuadrant-operator-controller-manager-86878c769f-wj4wc\" (UID: \"00c92151-a9bb-4bae-9040-aaf58be425f0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.688320 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.688242    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-wnd75\""
Jul 24 20:35:25.695915 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.695894    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:25.834104 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.834078    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"]
Jul 24 20:35:25.836169 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:35:25.836143    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00c92151_a9bb_4bae_9040_aaf58be425f0.slice/crio-f45ca07934692de75d586b1f190127f57b588e6583d725e11b0ffd730e34ef0e WatchSource:0}: Error finding container f45ca07934692de75d586b1f190127f57b588e6583d725e11b0ffd730e34ef0e: Status 404 returned error can't find the container with id f45ca07934692de75d586b1f190127f57b588e6583d725e11b0ffd730e34ef0e
Jul 24 20:35:25.894649 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:25.894622    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc" event={"ID":"00c92151-a9bb-4bae-9040-aaf58be425f0","Type":"ContainerStarted","Data":"f45ca07934692de75d586b1f190127f57b588e6583d725e11b0ffd730e34ef0e"}
Jul 24 20:35:26.900913 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:26.900881    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc" event={"ID":"00c92151-a9bb-4bae-9040-aaf58be425f0","Type":"ContainerStarted","Data":"f353aac5669c24ea2c64e24e8dcd98a6df8b3f6a56d40ea6b4a0391ba6fd307a"}
Jul 24 20:35:26.901293 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:26.901001    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:35:26.923575 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:26.923497    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc" podStartSLOduration=1.9234793909999999 podStartE2EDuration="1.923479391s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:35:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:35:26.922058759 +0000 UTC m=+1201.848197664" watchObservedRunningTime="2026-07-24 20:35:26.923479391 +0000 UTC m=+1201.849618299"
Jul 24 20:35:37.908493 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:35:37.908456    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-86878c769f-wj4wc"
Jul 24 20:38:12.394155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.394123    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"]
Jul 24 20:38:12.396654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.396633    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.399344 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.399325    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-eb87ecdf-kserve-self-signed-certs\""
Jul 24 20:38:12.415981 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.415960    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"]
Jul 24 20:38:12.506097 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.506064    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ckrhm\" (UniqueName: \"kubernetes.io/projected/f0fd3762-8452-4a07-9458-13883238c5fc-kube-api-access-ckrhm\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.506374 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.506106    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-home\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.506374 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.506129    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fd3762-8452-4a07-9458-13883238c5fc-tls-certs\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.506374 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.506206    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-dshm\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.506374 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.506282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-model-cache\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.506374 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.506311    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-tmp-dir\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607490 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607461    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-model-cache\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607503    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-tmp-dir\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607551    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ckrhm\" (UniqueName: \"kubernetes.io/projected/f0fd3762-8452-4a07-9458-13883238c5fc-kube-api-access-ckrhm\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607577    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-home\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607598    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fd3762-8452-4a07-9458-13883238c5fc-tls-certs\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607621    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-dshm\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607933 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607871    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-model-cache\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607972 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-tmp-dir\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.607972 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.607951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-home\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.609821 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.609797    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-dshm\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.610005 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.609987    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fd3762-8452-4a07-9458-13883238c5fc-tls-certs\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.615223 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.615202    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ckrhm\" (UniqueName: \"kubernetes.io/projected/f0fd3762-8452-4a07-9458-13883238c5fc-kube-api-access-ckrhm\") pod \"e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") " pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.707419 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.707353    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:12.833614 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:12.833585    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"]
Jul 24 20:38:12.835474 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:38:12.835446    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf0fd3762_8452_4a07_9458_13883238c5fc.slice/crio-917580bcdc4fe459ee774b6277f499ea9452195656266ab73a4f92247fec6733 WatchSource:0}: Error finding container 917580bcdc4fe459ee774b6277f499ea9452195656266ab73a4f92247fec6733: Status 404 returned error can't find the container with id 917580bcdc4fe459ee774b6277f499ea9452195656266ab73a4f92247fec6733
Jul 24 20:38:13.676766 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:13.676735    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr" event={"ID":"f0fd3762-8452-4a07-9458-13883238c5fc","Type":"ContainerStarted","Data":"c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c"}
Jul 24 20:38:13.677114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:13.676772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr" event={"ID":"f0fd3762-8452-4a07-9458-13883238c5fc","Type":"ContainerStarted","Data":"917580bcdc4fe459ee774b6277f499ea9452195656266ab73a4f92247fec6733"}
Jul 24 20:38:13.705230 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:13.705178    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr" podStartSLOduration=1.705163798 podStartE2EDuration="1.705163798s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:38:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:38:13.703508172 +0000 UTC m=+1368.629647076" watchObservedRunningTime="2026-07-24 20:38:13.705163798 +0000 UTC m=+1368.631302701"
Jul 24 20:38:20.177227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.177185    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"]
Jul 24 20:38:20.179746 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.179729    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.182281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.182260    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-eb87ecdf-kserve-self-signed-certs\""
Jul 24 20:38:20.190239 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.190217    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"]
Jul 24 20:38:20.272907 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.272879    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tmp-dir\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.273012 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.272978    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-dshm\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.273070 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.273021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-model-cache\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.273070 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.273049    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-home\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.273139 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.273071    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tls-certs\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.273173 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.273127    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dlwrh\" (UniqueName: \"kubernetes.io/projected/dbeca28c-0715-485c-ad4a-9efd9a6c021c-kube-api-access-dlwrh\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.373911 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.373884    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tmp-dir\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374053 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.373955    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-dshm\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374053 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.373999    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-model-cache\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.374118    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-home\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.374155    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tls-certs\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374265 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.374207    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dlwrh\" (UniqueName: \"kubernetes.io/projected/dbeca28c-0715-485c-ad4a-9efd9a6c021c-kube-api-access-dlwrh\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.374318    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tmp-dir\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374399 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.374386    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-model-cache\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.374593 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.374568    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-home\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.376273 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.376242    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-dshm\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.376981 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.376964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tls-certs\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.385325 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.385306    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dlwrh\" (UniqueName: \"kubernetes.io/projected/dbeca28c-0715-485c-ad4a-9efd9a6c021c-kube-api-access-dlwrh\") pod \"e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") " pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.489946 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.489889    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:20.621671 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.621647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"]
Jul 24 20:38:20.623883 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:38:20.623853    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddbeca28c_0715_485c_ad4a_9efd9a6c021c.slice/crio-802fb1f7447ef9126b8f06a98fc66229e553ae8a17e55e633ee03c8ab662cc51 WatchSource:0}: Error finding container 802fb1f7447ef9126b8f06a98fc66229e553ae8a17e55e633ee03c8ab662cc51: Status 404 returned error can't find the container with id 802fb1f7447ef9126b8f06a98fc66229e553ae8a17e55e633ee03c8ab662cc51
Jul 24 20:38:20.709976 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.709899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2" event={"ID":"dbeca28c-0715-485c-ad4a-9efd9a6c021c","Type":"ContainerStarted","Data":"a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f"}
Jul 24 20:38:20.709976 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.709940    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2" event={"ID":"dbeca28c-0715-485c-ad4a-9efd9a6c021c","Type":"ContainerStarted","Data":"802fb1f7447ef9126b8f06a98fc66229e553ae8a17e55e633ee03c8ab662cc51"}
Jul 24 20:38:20.728135 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:20.728096    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2" podStartSLOduration=0.728081203 podStartE2EDuration="728.081203ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:38:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:38:20.726175149 +0000 UTC m=+1375.652314055" watchObservedRunningTime="2026-07-24 20:38:20.728081203 +0000 UTC m=+1375.654220106"
Jul 24 20:38:22.708225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:22.708154    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:22.708225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:22.708191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:22.720347 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:22.720325    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:22.730999 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:22.730980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:30.490810 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:30.490779    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:30.491282 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:30.490823    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:30.503152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:30.503125    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:30.767086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:30.767022    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:43.075156 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:43.075120    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"]
Jul 24 20:38:44.574209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:44.574173    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"]
Jul 24 20:38:58.095712 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.095669    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr" podUID="f0fd3762-8452-4a07-9458-13883238c5fc" containerName="main" containerID="cri-o://c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c" gracePeriod=45
Jul 24 20:38:58.348739 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.348717    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:58.468854 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.468833    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ckrhm\" (UniqueName: \"kubernetes.io/projected/f0fd3762-8452-4a07-9458-13883238c5fc-kube-api-access-ckrhm\") pod \"f0fd3762-8452-4a07-9458-13883238c5fc\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") "
Jul 24 20:38:58.468987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.468871    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-dshm\") pod \"f0fd3762-8452-4a07-9458-13883238c5fc\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") "
Jul 24 20:38:58.468987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.468900    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-tmp-dir\") pod \"f0fd3762-8452-4a07-9458-13883238c5fc\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") "
Jul 24 20:38:58.468987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.468920    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-home\") pod \"f0fd3762-8452-4a07-9458-13883238c5fc\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") "
Jul 24 20:38:58.468987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.468942    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-model-cache\") pod \"f0fd3762-8452-4a07-9458-13883238c5fc\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") "
Jul 24 20:38:58.468987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.468976    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fd3762-8452-4a07-9458-13883238c5fc-tls-certs\") pod \"f0fd3762-8452-4a07-9458-13883238c5fc\" (UID: \"f0fd3762-8452-4a07-9458-13883238c5fc\") "
Jul 24 20:38:58.469280 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.469120    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f0fd3762-8452-4a07-9458-13883238c5fc" (UID: "f0fd3762-8452-4a07-9458-13883238c5fc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:58.469280 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.469185    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-home" (OuterVolumeSpecName: "home") pod "f0fd3762-8452-4a07-9458-13883238c5fc" (UID: "f0fd3762-8452-4a07-9458-13883238c5fc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:58.469280 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.469222    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-model-cache" (OuterVolumeSpecName: "model-cache") pod "f0fd3762-8452-4a07-9458-13883238c5fc" (UID: "f0fd3762-8452-4a07-9458-13883238c5fc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:58.469401 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.469280    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-tmp-dir\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:58.469401 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.469292    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-home\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:58.469401 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.469303    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-model-cache\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:58.470995 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.470970    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0fd3762-8452-4a07-9458-13883238c5fc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f0fd3762-8452-4a07-9458-13883238c5fc" (UID: "f0fd3762-8452-4a07-9458-13883238c5fc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:38:58.471093 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.471024    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-dshm" (OuterVolumeSpecName: "dshm") pod "f0fd3762-8452-4a07-9458-13883238c5fc" (UID: "f0fd3762-8452-4a07-9458-13883238c5fc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:58.471093 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.471040    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0fd3762-8452-4a07-9458-13883238c5fc-kube-api-access-ckrhm" (OuterVolumeSpecName: "kube-api-access-ckrhm") pod "f0fd3762-8452-4a07-9458-13883238c5fc" (UID: "f0fd3762-8452-4a07-9458-13883238c5fc"). InnerVolumeSpecName "kube-api-access-ckrhm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:38:58.570277 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.570252    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f0fd3762-8452-4a07-9458-13883238c5fc-tls-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:58.570277 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.570272    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ckrhm\" (UniqueName: \"kubernetes.io/projected/f0fd3762-8452-4a07-9458-13883238c5fc-kube-api-access-ckrhm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:58.570277 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.570280    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f0fd3762-8452-4a07-9458-13883238c5fc-dshm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:58.878316 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.878284    2566 generic.go:356] "Generic (PLEG): container finished" podID="f0fd3762-8452-4a07-9458-13883238c5fc" containerID="c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c" exitCode=0
Jul 24 20:38:58.878460 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.878320    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"
Jul 24 20:38:58.878460 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.878367    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr" event={"ID":"f0fd3762-8452-4a07-9458-13883238c5fc","Type":"ContainerDied","Data":"c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c"}
Jul 24 20:38:58.878460 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.878411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr" event={"ID":"f0fd3762-8452-4a07-9458-13883238c5fc","Type":"ContainerDied","Data":"917580bcdc4fe459ee774b6277f499ea9452195656266ab73a4f92247fec6733"}
Jul 24 20:38:58.878460 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.878429    2566 scope.go:117] "RemoveContainer" containerID="c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c"
Jul 24 20:38:58.887537 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.887507    2566 scope.go:117] "RemoveContainer" containerID="c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c"
Jul 24 20:38:58.887767 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:38:58.887751    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c\": container with ID starting with c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c not found: ID does not exist" containerID="c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c"
Jul 24 20:38:58.887827 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.887777    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c"} err="failed to get container status \"c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c\": rpc error: code = NotFound desc = could not find container \"c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c\": container with ID starting with c66e77ae1ccaa5bf765a06dbfee54701ef34fc83a624a2689fb0836830d6a71c not found: ID does not exist"
Jul 24 20:38:58.898859 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.898833    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"]
Jul 24 20:38:58.902542 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:58.902512    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-eb87ecdf-kserve-7b7595574c-pz4kr"]
Jul 24 20:38:59.593949 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.593900    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2" podUID="dbeca28c-0715-485c-ad4a-9efd9a6c021c" containerName="main" containerID="cri-o://a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f" gracePeriod=45
Jul 24 20:38:59.697154 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.697126    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0fd3762-8452-4a07-9458-13883238c5fc" path="/var/lib/kubelet/pods/f0fd3762-8452-4a07-9458-13883238c5fc/volumes"
Jul 24 20:38:59.846697 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.846630    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:59.886281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.886253    2566 generic.go:356] "Generic (PLEG): container finished" podID="dbeca28c-0715-485c-ad4a-9efd9a6c021c" containerID="a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f" exitCode=0
Jul 24 20:38:59.886398 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.886296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"
Jul 24 20:38:59.886398 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.886334    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2" event={"ID":"dbeca28c-0715-485c-ad4a-9efd9a6c021c","Type":"ContainerDied","Data":"a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f"}
Jul 24 20:38:59.886398 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.886374    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2" event={"ID":"dbeca28c-0715-485c-ad4a-9efd9a6c021c","Type":"ContainerDied","Data":"802fb1f7447ef9126b8f06a98fc66229e553ae8a17e55e633ee03c8ab662cc51"}
Jul 24 20:38:59.886398 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.886390    2566 scope.go:117] "RemoveContainer" containerID="a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f"
Jul 24 20:38:59.895665 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.895647    2566 scope.go:117] "RemoveContainer" containerID="a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f"
Jul 24 20:38:59.895930 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:38:59.895914    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f\": container with ID starting with a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f not found: ID does not exist" containerID="a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f"
Jul 24 20:38:59.895987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.895937    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f"} err="failed to get container status \"a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f\": rpc error: code = NotFound desc = could not find container \"a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f\": container with ID starting with a6c1f5466b97d6638c6f264723a858b7aebc120ef2b12303b871fab63547eb6f not found: ID does not exist"
Jul 24 20:38:59.981755 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.981735    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tls-certs\") pod \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") "
Jul 24 20:38:59.981861 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.981784    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-model-cache\") pod \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") "
Jul 24 20:38:59.981861 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.981803    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tmp-dir\") pod \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") "
Jul 24 20:38:59.981861 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.981829    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dlwrh\" (UniqueName: \"kubernetes.io/projected/dbeca28c-0715-485c-ad4a-9efd9a6c021c-kube-api-access-dlwrh\") pod \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") "
Jul 24 20:38:59.982026 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.981918    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-home\") pod \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") "
Jul 24 20:38:59.982026 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.981941    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-dshm\") pod \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\" (UID: \"dbeca28c-0715-485c-ad4a-9efd9a6c021c\") "
Jul 24 20:38:59.982026 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.981964    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-model-cache" (OuterVolumeSpecName: "model-cache") pod "dbeca28c-0715-485c-ad4a-9efd9a6c021c" (UID: "dbeca28c-0715-485c-ad4a-9efd9a6c021c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:59.982174 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.982076    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "dbeca28c-0715-485c-ad4a-9efd9a6c021c" (UID: "dbeca28c-0715-485c-ad4a-9efd9a6c021c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:59.982229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.982186    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-home" (OuterVolumeSpecName: "home") pod "dbeca28c-0715-485c-ad4a-9efd9a6c021c" (UID: "dbeca28c-0715-485c-ad4a-9efd9a6c021c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:59.982229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.982197    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-model-cache\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:59.982229 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.982215    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tmp-dir\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:38:59.983711 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.983684    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "dbeca28c-0715-485c-ad4a-9efd9a6c021c" (UID: "dbeca28c-0715-485c-ad4a-9efd9a6c021c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:38:59.983954 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.983930    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-dshm" (OuterVolumeSpecName: "dshm") pod "dbeca28c-0715-485c-ad4a-9efd9a6c021c" (UID: "dbeca28c-0715-485c-ad4a-9efd9a6c021c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:38:59.983954 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:38:59.983945    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbeca28c-0715-485c-ad4a-9efd9a6c021c-kube-api-access-dlwrh" (OuterVolumeSpecName: "kube-api-access-dlwrh") pod "dbeca28c-0715-485c-ad4a-9efd9a6c021c" (UID: "dbeca28c-0715-485c-ad4a-9efd9a6c021c"). InnerVolumeSpecName "kube-api-access-dlwrh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:39:00.083021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:39:00.082999    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-home\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:39:00.083021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:39:00.083022    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dbeca28c-0715-485c-ad4a-9efd9a6c021c-dshm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:39:00.083143 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:39:00.083035    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dbeca28c-0715-485c-ad4a-9efd9a6c021c-tls-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:39:00.083143 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:39:00.083048    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dlwrh\" (UniqueName: \"kubernetes.io/projected/dbeca28c-0715-485c-ad4a-9efd9a6c021c-kube-api-access-dlwrh\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:39:00.206332 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:39:00.206299    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"]
Jul 24 20:39:00.208609 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:39:00.208580    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-eb87ecdf-kserve-5858b8b8f5-xnwp2"]
Jul 24 20:39:01.693355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:39:01.693320    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dbeca28c-0715-485c-ad4a-9efd9a6c021c" path="/var/lib/kubelet/pods/dbeca28c-0715-485c-ad4a-9efd9a6c021c/volumes"
Jul 24 20:42:22.881377 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.881277    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"]
Jul 24 20:42:22.884086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.881680    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0fd3762-8452-4a07-9458-13883238c5fc" containerName="main"
Jul 24 20:42:22.884086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.881693    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0fd3762-8452-4a07-9458-13883238c5fc" containerName="main"
Jul 24 20:42:22.884086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.881712    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dbeca28c-0715-485c-ad4a-9efd9a6c021c" containerName="main"
Jul 24 20:42:22.884086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.881719    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="dbeca28c-0715-485c-ad4a-9efd9a6c021c" containerName="main"
Jul 24 20:42:22.884086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.881781    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="dbeca28c-0715-485c-ad4a-9efd9a6c021c" containerName="main"
Jul 24 20:42:22.884086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.881789    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0fd3762-8452-4a07-9458-13883238c5fc" containerName="main"
Jul 24 20:42:22.884998 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.884979    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:22.887818 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.887794    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-8c1aeab3-serving-cert\""
Jul 24 20:42:22.894383 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.894360    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"]
Jul 24 20:42:22.987242 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.987213    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-openshift-service-ca\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:22.987419 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.987280    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nbwn\" (UniqueName: \"kubernetes.io/projected/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-kube-api-access-6nbwn\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:22.987419 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:22.987304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-maas-api-tls\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.088617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.088588    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nbwn\" (UniqueName: \"kubernetes.io/projected/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-kube-api-access-6nbwn\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.088785 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.088631    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-maas-api-tls\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.088785 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.088726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-openshift-service-ca\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.089351 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.089328    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-openshift-service-ca\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.090939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.090920    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-maas-api-tls\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.100832 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.100809    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nbwn\" (UniqueName: \"kubernetes.io/projected/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-kube-api-access-6nbwn\") pod \"maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.196468 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.196416    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:23.532880 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.532854    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"]
Jul 24 20:42:23.534478 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:42:23.534448    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod78fb7a1b_3aca_45bc_9dc4_6ee189d10539.slice/crio-504566cc54641a67b8e1d277b3ba961451a51e40c1e26f75df58042fddcf08da WatchSource:0}: Error finding container 504566cc54641a67b8e1d277b3ba961451a51e40c1e26f75df58042fddcf08da: Status 404 returned error can't find the container with id 504566cc54641a67b8e1d277b3ba961451a51e40c1e26f75df58042fddcf08da
Jul 24 20:42:23.536136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.536112    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 20:42:23.593685 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.593659    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"]
Jul 24 20:42:23.598184 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.598168    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:23.601349 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.601312    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-8c1aeab3\""
Jul 24 20:42:23.601485 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.601312    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-8c1aeab3-dockercfg-trf7c\""
Jul 24 20:42:23.608214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.608191    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"]
Jul 24 20:42:23.693779 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.693747    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7668\" (UniqueName: \"kubernetes.io/projected/5b30d82e-628c-4b5b-9348-a740fe2f5142-kube-api-access-z7668\") pod \"payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:23.693927 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.693832    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b30d82e-628c-4b5b-9348-a740fe2f5142-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:23.792236 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.792156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n" event={"ID":"78fb7a1b-3aca-45bc-9dc4-6ee189d10539","Type":"ContainerStarted","Data":"504566cc54641a67b8e1d277b3ba961451a51e40c1e26f75df58042fddcf08da"}
Jul 24 20:42:23.794640 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.794620    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7668\" (UniqueName: \"kubernetes.io/projected/5b30d82e-628c-4b5b-9348-a740fe2f5142-kube-api-access-z7668\") pod \"payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:23.794709 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.794667    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b30d82e-628c-4b5b-9348-a740fe2f5142-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:23.795442 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.795414    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b30d82e-628c-4b5b-9348-a740fe2f5142-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:23.802589 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.802570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7668\" (UniqueName: \"kubernetes.io/projected/5b30d82e-628c-4b5b-9348-a740fe2f5142-kube-api-access-z7668\") pod \"payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:23.909988 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:23.909960    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:24.038682 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.038647    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"]
Jul 24 20:42:24.049745 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:42:24.049720    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b30d82e_628c_4b5b_9348_a740fe2f5142.slice/crio-10f6ae240df76872bcc82820d74f00256e51049c0e5b7482d21d2d763641c5c6 WatchSource:0}: Error finding container 10f6ae240df76872bcc82820d74f00256e51049c0e5b7482d21d2d763641c5c6: Status 404 returned error can't find the container with id 10f6ae240df76872bcc82820d74f00256e51049c0e5b7482d21d2d763641c5c6
Jul 24 20:42:24.798552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.798489    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n" event={"ID":"78fb7a1b-3aca-45bc-9dc4-6ee189d10539","Type":"ContainerStarted","Data":"048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15"}
Jul 24 20:42:24.798753 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.798670    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:24.800106 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.800082    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz" event={"ID":"5b30d82e-628c-4b5b-9348-a740fe2f5142","Type":"ContainerStarted","Data":"f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461"}
Jul 24 20:42:24.800249 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.800117    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz" event={"ID":"5b30d82e-628c-4b5b-9348-a740fe2f5142","Type":"ContainerStarted","Data":"10f6ae240df76872bcc82820d74f00256e51049c0e5b7482d21d2d763641c5c6"}
Jul 24 20:42:24.800249 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.800213    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:24.815795 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.815751    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n" podStartSLOduration=2.338535843 podStartE2EDuration="2.81573696s" totalImagesPullingTime="477.201117ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:42:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:42:24.813750395 +0000 UTC m=+1619.739889300" watchObservedRunningTime="2026-07-24 20:42:24.81573696 +0000 UTC m=+1619.741875863"
Jul 24 20:42:24.828704 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:24.828663    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz" podStartSLOduration=1.8286512990000001 podStartE2EDuration="1.828651299s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:42:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:42:24.828202682 +0000 UTC m=+1619.754341615" watchObservedRunningTime="2026-07-24 20:42:24.828651299 +0000 UTC m=+1619.754790203"
Jul 24 20:42:30.811223 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:30.811146    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:35.807929 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:35.807899    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:48.610725 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.610691    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"]
Jul 24 20:42:48.614474 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.614457    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"
Jul 24 20:42:48.617027 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.617006    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 20:42:48.622030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.621706    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"]
Jul 24 20:42:48.711713 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.711679    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb8ae60-ed5a-4896-8c84-b1d838450c8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9\" (UID: \"afb8ae60-ed5a-4896-8c84-b1d838450c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"
Jul 24 20:42:48.813112 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.813067    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb8ae60-ed5a-4896-8c84-b1d838450c8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9\" (UID: \"afb8ae60-ed5a-4896-8c84-b1d838450c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"
Jul 24 20:42:48.813743 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.813717    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb8ae60-ed5a-4896-8c84-b1d838450c8b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9\" (UID: \"afb8ae60-ed5a-4896-8c84-b1d838450c8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"
Jul 24 20:42:48.927305 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:48.927224    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"
Jul 24 20:42:49.060174 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:49.060149    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"]
Jul 24 20:42:49.061247 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:42:49.061207    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podafb8ae60_ed5a_4896_8c84_b1d838450c8b.slice/crio-788138c098ea6ec07ef3f144ccf8ad1e8697434b60771b1f516f06f3e2771621 WatchSource:0}: Error finding container 788138c098ea6ec07ef3f144ccf8ad1e8697434b60771b1f516f06f3e2771621: Status 404 returned error can't find the container with id 788138c098ea6ec07ef3f144ccf8ad1e8697434b60771b1f516f06f3e2771621
Jul 24 20:42:49.917086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:49.917045    2566 generic.go:356] "Generic (PLEG): container finished" podID="afb8ae60-ed5a-4896-8c84-b1d838450c8b" containerID="11cb9dfa67278fa33bb6c7ca84c743df72690091466c2bc1e19df4878c497d36" exitCode=0
Jul 24 20:42:49.917472 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:49.917109    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9" event={"ID":"afb8ae60-ed5a-4896-8c84-b1d838450c8b","Type":"ContainerDied","Data":"11cb9dfa67278fa33bb6c7ca84c743df72690091466c2bc1e19df4878c497d36"}
Jul 24 20:42:49.917472 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:49.917156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9" event={"ID":"afb8ae60-ed5a-4896-8c84-b1d838450c8b","Type":"ContainerStarted","Data":"788138c098ea6ec07ef3f144ccf8ad1e8697434b60771b1f516f06f3e2771621"}
Jul 24 20:42:51.061230 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:51.061203    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"
Jul 24 20:42:51.136196 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:51.136160    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb8ae60-ed5a-4896-8c84-b1d838450c8b-maas-api-service-ca\") pod \"afb8ae60-ed5a-4896-8c84-b1d838450c8b\" (UID: \"afb8ae60-ed5a-4896-8c84-b1d838450c8b\") "
Jul 24 20:42:51.136541 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:51.136502    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/afb8ae60-ed5a-4896-8c84-b1d838450c8b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "afb8ae60-ed5a-4896-8c84-b1d838450c8b" (UID: "afb8ae60-ed5a-4896-8c84-b1d838450c8b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:42:51.237679 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:51.237592    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/afb8ae60-ed5a-4896-8c84-b1d838450c8b-maas-api-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:42:51.928809 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:51.928782    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"
Jul 24 20:42:51.928809 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:51.928797    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9" event={"ID":"afb8ae60-ed5a-4896-8c84-b1d838450c8b","Type":"ContainerDied","Data":"788138c098ea6ec07ef3f144ccf8ad1e8697434b60771b1f516f06f3e2771621"}
Jul 24 20:42:51.929004 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:51.928842    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="788138c098ea6ec07ef3f144ccf8ad1e8697434b60771b1f516f06f3e2771621"
Jul 24 20:42:58.962247 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:58.962213    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"]
Jul 24 20:42:58.962922 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:58.962425    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n" podUID="78fb7a1b-3aca-45bc-9dc4-6ee189d10539" containerName="maas-api" containerID="cri-o://048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15" gracePeriod=30
Jul 24 20:42:59.010591 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.010556    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"]
Jul 24 20:42:59.010846 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.010802    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz" podUID="5b30d82e-628c-4b5b-9348-a740fe2f5142" containerName="payload-pre-processing" containerID="cri-o://f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461" gracePeriod=30
Jul 24 20:42:59.264402 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.264377    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:59.279153 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.279126    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:59.411478 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.411431    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b30d82e-628c-4b5b-9348-a740fe2f5142-plugins-config-volume\") pod \"5b30d82e-628c-4b5b-9348-a740fe2f5142\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") "
Jul 24 20:42:59.411675 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.411505    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z7668\" (UniqueName: \"kubernetes.io/projected/5b30d82e-628c-4b5b-9348-a740fe2f5142-kube-api-access-z7668\") pod \"5b30d82e-628c-4b5b-9348-a740fe2f5142\" (UID: \"5b30d82e-628c-4b5b-9348-a740fe2f5142\") "
Jul 24 20:42:59.411675 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.411564    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-maas-api-tls\") pod \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") "
Jul 24 20:42:59.411675 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.411601    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6nbwn\" (UniqueName: \"kubernetes.io/projected/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-kube-api-access-6nbwn\") pod \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") "
Jul 24 20:42:59.411675 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.411655    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-openshift-service-ca\") pod \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\" (UID: \"78fb7a1b-3aca-45bc-9dc4-6ee189d10539\") "
Jul 24 20:42:59.411883 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.411854    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b30d82e-628c-4b5b-9348-a740fe2f5142-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5b30d82e-628c-4b5b-9348-a740fe2f5142" (UID: "5b30d82e-628c-4b5b-9348-a740fe2f5142"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:42:59.411992 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.411969    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5b30d82e-628c-4b5b-9348-a740fe2f5142-plugins-config-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:42:59.412117 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.412090    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "78fb7a1b-3aca-45bc-9dc4-6ee189d10539" (UID: "78fb7a1b-3aca-45bc-9dc4-6ee189d10539"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:42:59.413680 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.413659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "78fb7a1b-3aca-45bc-9dc4-6ee189d10539" (UID: "78fb7a1b-3aca-45bc-9dc4-6ee189d10539"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:42:59.414021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.414000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b30d82e-628c-4b5b-9348-a740fe2f5142-kube-api-access-z7668" (OuterVolumeSpecName: "kube-api-access-z7668") pod "5b30d82e-628c-4b5b-9348-a740fe2f5142" (UID: "5b30d82e-628c-4b5b-9348-a740fe2f5142"). InnerVolumeSpecName "kube-api-access-z7668". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:42:59.414087 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.414028    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-kube-api-access-6nbwn" (OuterVolumeSpecName: "kube-api-access-6nbwn") pod "78fb7a1b-3aca-45bc-9dc4-6ee189d10539" (UID: "78fb7a1b-3aca-45bc-9dc4-6ee189d10539"). InnerVolumeSpecName "kube-api-access-6nbwn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:42:59.512828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.512734    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:42:59.512828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.512774    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7668\" (UniqueName: \"kubernetes.io/projected/5b30d82e-628c-4b5b-9348-a740fe2f5142-kube-api-access-z7668\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:42:59.512828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.512784    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:42:59.512828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.512793    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nbwn\" (UniqueName: \"kubernetes.io/projected/78fb7a1b-3aca-45bc-9dc4-6ee189d10539-kube-api-access-6nbwn\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:42:59.975678 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.975629    2566 generic.go:356] "Generic (PLEG): container finished" podID="5b30d82e-628c-4b5b-9348-a740fe2f5142" containerID="f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461" exitCode=0
Jul 24 20:42:59.975678 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.975677    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"
Jul 24 20:42:59.976317 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.975724    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz" event={"ID":"5b30d82e-628c-4b5b-9348-a740fe2f5142","Type":"ContainerDied","Data":"f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461"}
Jul 24 20:42:59.976317 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.975778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz" event={"ID":"5b30d82e-628c-4b5b-9348-a740fe2f5142","Type":"ContainerDied","Data":"10f6ae240df76872bcc82820d74f00256e51049c0e5b7482d21d2d763641c5c6"}
Jul 24 20:42:59.976317 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.975806    2566 scope.go:117] "RemoveContainer" containerID="f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461"
Jul 24 20:42:59.977357 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.977338    2566 generic.go:356] "Generic (PLEG): container finished" podID="78fb7a1b-3aca-45bc-9dc4-6ee189d10539" containerID="048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15" exitCode=0
Jul 24 20:42:59.977448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.977363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n" event={"ID":"78fb7a1b-3aca-45bc-9dc4-6ee189d10539","Type":"ContainerDied","Data":"048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15"}
Jul 24 20:42:59.977448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.977394    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"
Jul 24 20:42:59.977448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.977424    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n" event={"ID":"78fb7a1b-3aca-45bc-9dc4-6ee189d10539","Type":"ContainerDied","Data":"504566cc54641a67b8e1d277b3ba961451a51e40c1e26f75df58042fddcf08da"}
Jul 24 20:42:59.991879 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.991864    2566 scope.go:117] "RemoveContainer" containerID="f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461"
Jul 24 20:42:59.992195 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:42:59.992174    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461\": container with ID starting with f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461 not found: ID does not exist" containerID="f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461"
Jul 24 20:42:59.992263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.992203    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461"} err="failed to get container status \"f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461\": rpc error: code = NotFound desc = could not find container \"f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461\": container with ID starting with f2533cb80e945b0297eb8b97c0099eb2f60d659794d160315193d7e6ea187461 not found: ID does not exist"
Jul 24 20:42:59.992263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:42:59.992221    2566 scope.go:117] "RemoveContainer" containerID="048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15"
Jul 24 20:43:00.001204 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:00.001182    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"]
Jul 24 20:43:00.002352 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:00.002338    2566 scope.go:117] "RemoveContainer" containerID="048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15"
Jul 24 20:43:00.002656 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:43:00.002631    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15\": container with ID starting with 048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15 not found: ID does not exist" containerID="048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15"
Jul 24 20:43:00.002733 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:00.002660    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15"} err="failed to get container status \"048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15\": rpc error: code = NotFound desc = could not find container \"048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15\": container with ID starting with 048eabd138d246ae43f8419d5e4f76ab02fbe6b1842ce3efaa7d74cbfcf6ca15 not found: ID does not exist"
Jul 24 20:43:00.007960 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:00.007931    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-8c1aeab3-7678b4d8d4-csjkz"]
Jul 24 20:43:00.015300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:00.015278    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"]
Jul 24 20:43:00.018949 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:00.018927    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-8c1aeab3-54d487f697-gwh5n"]
Jul 24 20:43:01.692066 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:01.692030    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b30d82e-628c-4b5b-9348-a740fe2f5142" path="/var/lib/kubelet/pods/5b30d82e-628c-4b5b-9348-a740fe2f5142/volumes"
Jul 24 20:43:01.692626 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:01.692608    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="78fb7a1b-3aca-45bc-9dc4-6ee189d10539" path="/var/lib/kubelet/pods/78fb7a1b-3aca-45bc-9dc4-6ee189d10539/volumes"
Jul 24 20:43:03.678417 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:03.678382    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"]
Jul 24 20:43:03.681039 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:03.681014    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-8c1aeab3-bfqf9"]
Jul 24 20:43:03.692372 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:03.692347    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="afb8ae60-ed5a-4896-8c84-b1d838450c8b" path="/var/lib/kubelet/pods/afb8ae60-ed5a-4896-8c84-b1d838450c8b/volumes"
Jul 24 20:43:04.914654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.914619    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"]
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.914963    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b30d82e-628c-4b5b-9348-a740fe2f5142" containerName="payload-pre-processing"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.914973    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b30d82e-628c-4b5b-9348-a740fe2f5142" containerName="payload-pre-processing"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.914983    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="afb8ae60-ed5a-4896-8c84-b1d838450c8b" containerName="revoke-keys"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.914988    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="afb8ae60-ed5a-4896-8c84-b1d838450c8b" containerName="revoke-keys"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.915002    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="78fb7a1b-3aca-45bc-9dc4-6ee189d10539" containerName="maas-api"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.915008    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="78fb7a1b-3aca-45bc-9dc4-6ee189d10539" containerName="maas-api"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.915063    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="afb8ae60-ed5a-4896-8c84-b1d838450c8b" containerName="revoke-keys"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.915077    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b30d82e-628c-4b5b-9348-a740fe2f5142" containerName="payload-pre-processing"
Jul 24 20:43:04.915152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.915088    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="78fb7a1b-3aca-45bc-9dc4-6ee189d10539" containerName="maas-api"
Jul 24 20:43:04.919970 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.919946    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:04.923278 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.923255    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-42b8f369-gw-openshift-default-dockercfg-cb6gj\""
Jul 24 20:43:04.929539 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:04.929504    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"]
Jul 24 20:43:05.061909 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.061872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istiod-ca-cert\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062083 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.061921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-credential-socket\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062083 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.061939    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-data\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062083 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.061964    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh9zf\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-kube-api-access-qh9zf\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062083 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.062005    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-podinfo\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062223 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.062077    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-envoy\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062223 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.062116    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-certs\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062223 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.062152    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-socket\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.062223 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.062167    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-token\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.162937 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.162904    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-podinfo\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163121 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.162962    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-envoy\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163121 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.162989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-certs\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163121 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163024    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-socket\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163121 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163048    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-token\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163334 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163284    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istiod-ca-cert\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163389 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163354    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-credential-socket\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163444 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163391    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-data\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163444 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163418    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh9zf\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-kube-api-access-qh9zf\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163591 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163443    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-certs\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163591 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163480    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-socket\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163773 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163746    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-credential-socket\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.163899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.163779    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-data\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.164071 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.164049    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istiod-ca-cert\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.165240 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.165183    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-envoy\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.165520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.165503    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-podinfo\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.170862 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.170833    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-token\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.170967 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.170951    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh9zf\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-kube-api-access-qh9zf\") pod \"e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") " pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.230720 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.230684    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:05.370177 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.370153    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"]
Jul 24 20:43:05.372068 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:43:05.372029    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee8519fd_6ca3_4cfc_b996_16f764d5c638.slice/crio-066d24b19341d928085a0dfb51e07f54ea235ecf8a518c41fd340bef9fa9f359 WatchSource:0}: Error finding container 066d24b19341d928085a0dfb51e07f54ea235ecf8a518c41fd340bef9fa9f359: Status 404 returned error can't find the container with id 066d24b19341d928085a0dfb51e07f54ea235ecf8a518c41fd340bef9fa9f359
Jul 24 20:43:05.374257 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.374227    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:43:05.374336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.374291    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:43:05.374336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:05.374318    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:43:06.007277 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:06.007242    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" event={"ID":"ee8519fd-6ca3-4cfc-b996-16f764d5c638","Type":"ContainerStarted","Data":"e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398"}
Jul 24 20:43:06.007696 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:06.007295    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" event={"ID":"ee8519fd-6ca3-4cfc-b996-16f764d5c638","Type":"ContainerStarted","Data":"066d24b19341d928085a0dfb51e07f54ea235ecf8a518c41fd340bef9fa9f359"}
Jul 24 20:43:06.032743 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:06.032693    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" podStartSLOduration=2.032679482 podStartE2EDuration="2.032679482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:43:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:43:06.031583538 +0000 UTC m=+1660.957722460" watchObservedRunningTime="2026-07-24 20:43:06.032679482 +0000 UTC m=+1660.958818386"
Jul 24 20:43:06.231389 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:06.231346    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:06.236202 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:06.236166    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:07.012400 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.012371    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:07.013690 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.013664    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:07.595458 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.595420    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"]
Jul 24 20:43:07.598995 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.598976    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.601794 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.601779    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-42b8f369-serving-cert\""
Jul 24 20:43:07.610001 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.609975    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"]
Jul 24 20:43:07.685723 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.685691    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0823af3-c470-4eba-aa44-cfa8076d3c69-openshift-service-ca\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.685723 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.685722    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.685925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.685750    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dtbv\" (UniqueName: \"kubernetes.io/projected/f0823af3-c470-4eba-aa44-cfa8076d3c69-kube-api-access-2dtbv\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.787136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.787103    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dtbv\" (UniqueName: \"kubernetes.io/projected/f0823af3-c470-4eba-aa44-cfa8076d3c69-kube-api-access-2dtbv\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.787310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.787189    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0823af3-c470-4eba-aa44-cfa8076d3c69-openshift-service-ca\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.787310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.787209    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.787310 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:43:07.787303    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-serving-cert: secret "maas-api-e2e-migrate-42b8f369-serving-cert" not found
Jul 24 20:43:07.787417 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:43:07.787384    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls podName:f0823af3-c470-4eba-aa44-cfa8076d3c69 nodeName:}" failed. No retries permitted until 2026-07-24 20:43:08.287363686 +0000 UTC m=+1663.213502569 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls") pod "maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z" (UID: "f0823af3-c470-4eba-aa44-cfa8076d3c69") : secret "maas-api-e2e-migrate-42b8f369-serving-cert" not found
Jul 24 20:43:07.787931 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.787911    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0823af3-c470-4eba-aa44-cfa8076d3c69-openshift-service-ca\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:07.796546 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:07.796499    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dtbv\" (UniqueName: \"kubernetes.io/projected/f0823af3-c470-4eba-aa44-cfa8076d3c69-kube-api-access-2dtbv\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:08.292680 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:08.292645    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:08.295133 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:08.295108    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls\") pod \"maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:08.510059 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:08.510023    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:08.653672 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:08.653636    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"]
Jul 24 20:43:08.656199 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:43:08.656174    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0823af3_c470_4eba_aa44_cfa8076d3c69.slice/crio-ee42238a954fcd27313108fc5d2e3a34f8fda017eb174fcdf914d359b65bd84b WatchSource:0}: Error finding container ee42238a954fcd27313108fc5d2e3a34f8fda017eb174fcdf914d359b65bd84b: Status 404 returned error can't find the container with id ee42238a954fcd27313108fc5d2e3a34f8fda017eb174fcdf914d359b65bd84b
Jul 24 20:43:09.024356 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:09.024325    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z" event={"ID":"f0823af3-c470-4eba-aa44-cfa8076d3c69","Type":"ContainerStarted","Data":"ee42238a954fcd27313108fc5d2e3a34f8fda017eb174fcdf914d359b65bd84b"}
Jul 24 20:43:09.042601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:09.042540    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z" podStartSLOduration=1.743258495 podStartE2EDuration="2.042506374s" totalImagesPullingTime="299.247879ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:43:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:43:09.039139343 +0000 UTC m=+1663.965278250" watchObservedRunningTime="2026-07-24 20:43:09.042506374 +0000 UTC m=+1663.968645279"
Jul 24 20:43:10.030908 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:10.030866    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z" event={"ID":"f0823af3-c470-4eba-aa44-cfa8076d3c69","Type":"ContainerStarted","Data":"82c34cd9a9de09139e65a11433d65cdb72a5470c62dc90b0076b316b19be25ec"}
Jul 24 20:43:10.031302 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:10.031030    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:16.052886 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:16.052852    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:22.013950 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.013905    2566 patch_prober.go:28] interesting pod/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 20:43:22.014357 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.013983    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 20:43:22.094956 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.094923    2566 generic.go:356] "Generic (PLEG): container finished" podID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerID="e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398" exitCode=0
Jul 24 20:43:22.095155 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.094974    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" event={"ID":"ee8519fd-6ca3-4cfc-b996-16f764d5c638","Type":"ContainerDied","Data":"e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398"}
Jul 24 20:43:22.095420 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.095401    2566 scope.go:117] "RemoveContainer" containerID="e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398"
Jul 24 20:43:22.096432 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.096370    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:43:22.096512 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.096435    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:43:22.096512 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:22.096467    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:43:23.101065 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:23.101025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" event={"ID":"ee8519fd-6ca3-4cfc-b996-16f764d5c638","Type":"ContainerStarted","Data":"4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733"}
Jul 24 20:43:23.230871 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:23.230833    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:23.231049 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:23.231024    2566 patch_prober.go:28] interesting pod/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 20:43:23.231093 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:23.231073    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 20:43:24.231172 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:24.231139    2566 patch_prober.go:28] interesting pod/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 20:43:24.231622 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:24.231191    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 20:43:25.231746 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:25.231712    2566 patch_prober.go:28] interesting pod/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 20:43:25.232134 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:25.231768    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 20:43:26.231538 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:26.231502    2566 patch_prober.go:28] interesting pod/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused" start-of-body=
Jul 24 20:43:26.231709 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:26.231595    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.56:15021/healthz/ready\": dial tcp 10.133.0.56:15021: connect: connection refused"
Jul 24 20:43:27.235243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:27.235216    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:27.235628 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:27.235460    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:27.236089 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:27.236070    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:43.025642 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.025604    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"]
Jul 24 20:43:43.027240 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.025912    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z" podUID="f0823af3-c470-4eba-aa44-cfa8076d3c69" containerName="maas-api" containerID="cri-o://82c34cd9a9de09139e65a11433d65cdb72a5470c62dc90b0076b316b19be25ec" gracePeriod=30
Jul 24 20:43:43.194208 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.194174    2566 generic.go:356] "Generic (PLEG): container finished" podID="f0823af3-c470-4eba-aa44-cfa8076d3c69" containerID="82c34cd9a9de09139e65a11433d65cdb72a5470c62dc90b0076b316b19be25ec" exitCode=0
Jul 24 20:43:43.194372 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.194208    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z" event={"ID":"f0823af3-c470-4eba-aa44-cfa8076d3c69","Type":"ContainerDied","Data":"82c34cd9a9de09139e65a11433d65cdb72a5470c62dc90b0076b316b19be25ec"}
Jul 24 20:43:43.284723 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.284665    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:43.388411 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.388384    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dtbv\" (UniqueName: \"kubernetes.io/projected/f0823af3-c470-4eba-aa44-cfa8076d3c69-kube-api-access-2dtbv\") pod \"f0823af3-c470-4eba-aa44-cfa8076d3c69\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") "
Jul 24 20:43:43.388600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.388481    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0823af3-c470-4eba-aa44-cfa8076d3c69-openshift-service-ca\") pod \"f0823af3-c470-4eba-aa44-cfa8076d3c69\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") "
Jul 24 20:43:43.388600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.388510    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls\") pod \"f0823af3-c470-4eba-aa44-cfa8076d3c69\" (UID: \"f0823af3-c470-4eba-aa44-cfa8076d3c69\") "
Jul 24 20:43:43.388919 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.388888    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0823af3-c470-4eba-aa44-cfa8076d3c69-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "f0823af3-c470-4eba-aa44-cfa8076d3c69" (UID: "f0823af3-c470-4eba-aa44-cfa8076d3c69"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:43:43.390496 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.390466    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "f0823af3-c470-4eba-aa44-cfa8076d3c69" (UID: "f0823af3-c470-4eba-aa44-cfa8076d3c69"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:43:43.390814 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.390787    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0823af3-c470-4eba-aa44-cfa8076d3c69-kube-api-access-2dtbv" (OuterVolumeSpecName: "kube-api-access-2dtbv") pod "f0823af3-c470-4eba-aa44-cfa8076d3c69" (UID: "f0823af3-c470-4eba-aa44-cfa8076d3c69"). InnerVolumeSpecName "kube-api-access-2dtbv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:43:43.489866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.489835    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0823af3-c470-4eba-aa44-cfa8076d3c69-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:43.489866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.489860    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/f0823af3-c470-4eba-aa44-cfa8076d3c69-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:43.489866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:43.489869    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dtbv\" (UniqueName: \"kubernetes.io/projected/f0823af3-c470-4eba-aa44-cfa8076d3c69-kube-api-access-2dtbv\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:44.199513 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:44.199485    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"
Jul 24 20:43:44.199923 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:44.199487    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z" event={"ID":"f0823af3-c470-4eba-aa44-cfa8076d3c69","Type":"ContainerDied","Data":"ee42238a954fcd27313108fc5d2e3a34f8fda017eb174fcdf914d359b65bd84b"}
Jul 24 20:43:44.199923 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:44.199624    2566 scope.go:117] "RemoveContainer" containerID="82c34cd9a9de09139e65a11433d65cdb72a5470c62dc90b0076b316b19be25ec"
Jul 24 20:43:44.215004 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:44.214980    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"]
Jul 24 20:43:44.218386 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:44.218365    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-42b8f369-6754674cbd-rvf6z"]
Jul 24 20:43:45.692170 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:45.692140    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f0823af3-c470-4eba-aa44-cfa8076d3c69" path="/var/lib/kubelet/pods/f0823af3-c470-4eba-aa44-cfa8076d3c69/volumes"
Jul 24 20:43:47.811832 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:47.811747    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"]
Jul 24 20:43:47.812209 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:47.812029    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy" containerID="cri-o://4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733" gracePeriod=30
Jul 24 20:43:51.288146 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.288113    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"]
Jul 24 20:43:51.288688 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.288654    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f0823af3-c470-4eba-aa44-cfa8076d3c69" containerName="maas-api"
Jul 24 20:43:51.288688 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.288673    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f0823af3-c470-4eba-aa44-cfa8076d3c69" containerName="maas-api"
Jul 24 20:43:51.288811 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.288771    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f0823af3-c470-4eba-aa44-cfa8076d3c69" containerName="maas-api"
Jul 24 20:43:51.293113 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.293090    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.296478 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.296459    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-bd4a88b3-serving-cert\""
Jul 24 20:43:51.302261 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.302239    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"]
Jul 24 20:43:51.354015 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.353989    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfh2c\" (UniqueName: \"kubernetes.io/projected/32c4d263-879e-46b3-a4e3-7d7a313577ec-kube-api-access-qfh2c\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.354118 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.354021    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.354118 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.354050    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c4d263-879e-46b3-a4e3-7d7a313577ec-openshift-service-ca\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.454714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.454682    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfh2c\" (UniqueName: \"kubernetes.io/projected/32c4d263-879e-46b3-a4e3-7d7a313577ec-kube-api-access-qfh2c\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.454714 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.454718    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.454901 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.454767    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c4d263-879e-46b3-a4e3-7d7a313577ec-openshift-service-ca\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.454992 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:43:51.454971    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-serving-cert: secret "maas-api-e2e-ait-bd4a88b3-serving-cert" not found
Jul 24 20:43:51.455065 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:43:51.455055    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls podName:32c4d263-879e-46b3-a4e3-7d7a313577ec nodeName:}" failed. No retries permitted until 2026-07-24 20:43:51.955033222 +0000 UTC m=+1706.881172115 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls") pod "maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm" (UID: "32c4d263-879e-46b3-a4e3-7d7a313577ec") : secret "maas-api-e2e-ait-bd4a88b3-serving-cert" not found
Jul 24 20:43:51.455342 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.455326    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c4d263-879e-46b3-a4e3-7d7a313577ec-openshift-service-ca\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.469399 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.469370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfh2c\" (UniqueName: \"kubernetes.io/projected/32c4d263-879e-46b3-a4e3-7d7a313577ec-kube-api-access-qfh2c\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.959156 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.959117    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:51.961373 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:51.961350    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls\") pod \"maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:52.204211 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:52.204183    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:52.555199 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:52.555175    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"]
Jul 24 20:43:52.556911 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:43:52.556881    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32c4d263_879e_46b3_a4e3_7d7a313577ec.slice/crio-8f298f6447f37c03cf17cb559a10c1261886e321ac039d31fcd459011db0b395 WatchSource:0}: Error finding container 8f298f6447f37c03cf17cb559a10c1261886e321ac039d31fcd459011db0b395: Status 404 returned error can't find the container with id 8f298f6447f37c03cf17cb559a10c1261886e321ac039d31fcd459011db0b395
Jul 24 20:43:53.077448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.077426    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:53.167247 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167216    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-credential-socket\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-socket\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167302    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-data\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167340    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-envoy\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167361    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-token\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167413    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qh9zf\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-kube-api-access-qh9zf\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167486    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-certs\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167498    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:43:53.167555 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167509    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istiod-ca-cert\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167585    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:43:53.167991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167606    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-podinfo\") pod \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\" (UID: \"ee8519fd-6ca3-4cfc-b996-16f764d5c638\") "
Jul 24 20:43:53.167991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167783    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:43:53.167991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167917    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:43:53.167991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167958    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.167991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167977    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-credential-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.167991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.167991    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-workload-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.168274 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.168237    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-data" (OuterVolumeSpecName: "istio-data") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:43:53.169681 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.169653    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-token" (OuterVolumeSpecName: "istio-token") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:43:53.169783 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.169699    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 20:43:53.169828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.169795    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:43:53.169828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.169794    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-kube-api-access-qh9zf" (OuterVolumeSpecName: "kube-api-access-qh9zf") pod "ee8519fd-6ca3-4cfc-b996-16f764d5c638" (UID: "ee8519fd-6ca3-4cfc-b996-16f764d5c638"). InnerVolumeSpecName "kube-api-access-qh9zf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:43:53.248793 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.248761    2566 generic.go:356] "Generic (PLEG): container finished" podID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerID="4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733" exitCode=0
Jul 24 20:43:53.248939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.248831    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"
Jul 24 20:43:53.248939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.248851    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" event={"ID":"ee8519fd-6ca3-4cfc-b996-16f764d5c638","Type":"ContainerDied","Data":"4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733"}
Jul 24 20:43:53.248939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.248927    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d" event={"ID":"ee8519fd-6ca3-4cfc-b996-16f764d5c638","Type":"ContainerDied","Data":"066d24b19341d928085a0dfb51e07f54ea235ecf8a518c41fd340bef9fa9f359"}
Jul 24 20:43:53.249114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.248962    2566 scope.go:117] "RemoveContainer" containerID="4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733"
Jul 24 20:43:53.251135 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.251105    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm" event={"ID":"32c4d263-879e-46b3-a4e3-7d7a313577ec","Type":"ContainerStarted","Data":"f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8"}
Jul 24 20:43:53.251254 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.251156    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm" event={"ID":"32c4d263-879e-46b3-a4e3-7d7a313577ec","Type":"ContainerStarted","Data":"8f298f6447f37c03cf17cb559a10c1261886e321ac039d31fcd459011db0b395"}
Jul 24 20:43:53.251254 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.251197    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:43:53.258002 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.257983    2566 scope.go:117] "RemoveContainer" containerID="e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398"
Jul 24 20:43:53.265468 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.265452    2566 scope.go:117] "RemoveContainer" containerID="4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733"
Jul 24 20:43:53.265733 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:43:53.265716    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733\": container with ID starting with 4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733 not found: ID does not exist" containerID="4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733"
Jul 24 20:43:53.265789 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.265740    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733"} err="failed to get container status \"4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733\": rpc error: code = NotFound desc = could not find container \"4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733\": container with ID starting with 4d1a39901bb6f5d7cbc50d13959286af84aac66e7e65758773b98b811d012733 not found: ID does not exist"
Jul 24 20:43:53.265789 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.265758    2566 scope.go:117] "RemoveContainer" containerID="e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398"
Jul 24 20:43:53.265990 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:43:53.265974    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398\": container with ID starting with e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398 not found: ID does not exist" containerID="e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398"
Jul 24 20:43:53.266031 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.265996    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398"} err="failed to get container status \"e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398\": rpc error: code = NotFound desc = could not find container \"e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398\": container with ID starting with e910d155c4f9cc01eafc1bb3d745fc3a9e2b05e6368685747a8df670ce969398 not found: ID does not exist"
Jul 24 20:43:53.268334 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.268317    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istiod-ca-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.268418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.268336    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-podinfo\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.268418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.268346    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-data\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.268418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.268354    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-envoy\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.268418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.268361    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-istio-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.268418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.268368    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh9zf\" (UniqueName: \"kubernetes.io/projected/ee8519fd-6ca3-4cfc-b996-16f764d5c638-kube-api-access-qh9zf\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:43:53.268955 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.268920    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm" podStartSLOduration=1.818783284 podStartE2EDuration="2.268907988s" totalImagesPullingTime="450.124704ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:43:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:43:53.268407786 +0000 UTC m=+1708.194546692" watchObservedRunningTime="2026-07-24 20:43:53.268907988 +0000 UTC m=+1708.195046895"
Jul 24 20:43:53.282149 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.282126    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"]
Jul 24 20:43:53.285411 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.285393    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-42b8f369-gw-openshift-default-658dd45596-ccp9d"]
Jul 24 20:43:53.692665 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:53.692632    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" path="/var/lib/kubelet/pods/ee8519fd-6ca3-4cfc-b996-16f764d5c638/volumes"
Jul 24 20:43:59.262755 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:43:59.262722    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:44:17.268834 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.268802    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"]
Jul 24 20:44:17.269319 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.269136    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy"
Jul 24 20:44:17.269319 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.269147    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy"
Jul 24 20:44:17.269319 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.269218    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy"
Jul 24 20:44:17.269319 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.269226    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy"
Jul 24 20:44:17.272317 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.272299    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"
Jul 24 20:44:17.275225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.275201    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 20:44:17.282067 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.282046    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"]
Jul 24 20:44:17.453310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.453281    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a37fea-241e-41e5-923b-635cbb14fd77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6\" (UID: \"10a37fea-241e-41e5-923b-635cbb14fd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"
Jul 24 20:44:17.554025 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.553959    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a37fea-241e-41e5-923b-635cbb14fd77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6\" (UID: \"10a37fea-241e-41e5-923b-635cbb14fd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"
Jul 24 20:44:17.554540 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.554505    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a37fea-241e-41e5-923b-635cbb14fd77-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6\" (UID: \"10a37fea-241e-41e5-923b-635cbb14fd77\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"
Jul 24 20:44:17.583170 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.583148    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"
Jul 24 20:44:17.712021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:17.711999    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"]
Jul 24 20:44:17.713786 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:44:17.713755    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a37fea_241e_41e5_923b_635cbb14fd77.slice/crio-891e0ff7924c859a90fced34f7674edb865e528bd6face327ac0e26133753e4c WatchSource:0}: Error finding container 891e0ff7924c859a90fced34f7674edb865e528bd6face327ac0e26133753e4c: Status 404 returned error can't find the container with id 891e0ff7924c859a90fced34f7674edb865e528bd6face327ac0e26133753e4c
Jul 24 20:44:18.371907 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:18.371884    2566 generic.go:356] "Generic (PLEG): container finished" podID="10a37fea-241e-41e5-923b-635cbb14fd77" containerID="102628e874817f46a11d74e8dd0e02b6fbb37c407ace357be54f5fededf0c57e" exitCode=0
Jul 24 20:44:18.372206 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:18.371947    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6" event={"ID":"10a37fea-241e-41e5-923b-635cbb14fd77","Type":"ContainerDied","Data":"102628e874817f46a11d74e8dd0e02b6fbb37c407ace357be54f5fededf0c57e"}
Jul 24 20:44:18.372206 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:18.372002    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6" event={"ID":"10a37fea-241e-41e5-923b-635cbb14fd77","Type":"ContainerStarted","Data":"891e0ff7924c859a90fced34f7674edb865e528bd6face327ac0e26133753e4c"}
Jul 24 20:44:19.508404 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:19.508381    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"
Jul 24 20:44:19.673031 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:19.672965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a37fea-241e-41e5-923b-635cbb14fd77-maas-api-service-ca\") pod \"10a37fea-241e-41e5-923b-635cbb14fd77\" (UID: \"10a37fea-241e-41e5-923b-635cbb14fd77\") "
Jul 24 20:44:19.673300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:19.673278    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10a37fea-241e-41e5-923b-635cbb14fd77-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "10a37fea-241e-41e5-923b-635cbb14fd77" (UID: "10a37fea-241e-41e5-923b-635cbb14fd77"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:44:19.722992 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:44:19.722968    2566 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10a37fea_241e_41e5_923b_635cbb14fd77.slice\": RecentStats: unable to find data in memory cache]"
Jul 24 20:44:19.773802 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:19.773780    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/10a37fea-241e-41e5-923b-635cbb14fd77-maas-api-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:44:20.381309 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:20.381285    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"
Jul 24 20:44:20.381309 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:20.381301    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6" event={"ID":"10a37fea-241e-41e5-923b-635cbb14fd77","Type":"ContainerDied","Data":"891e0ff7924c859a90fced34f7674edb865e528bd6face327ac0e26133753e4c"}
Jul 24 20:44:20.381497 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:20.381341    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="891e0ff7924c859a90fced34f7674edb865e528bd6face327ac0e26133753e4c"
Jul 24 20:44:27.632236 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.632201    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"]
Jul 24 20:44:27.632610 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.632425    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm" podUID="32c4d263-879e-46b3-a4e3-7d7a313577ec" containerName="maas-api" containerID="cri-o://f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8" gracePeriod=30
Jul 24 20:44:27.878931 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.878908    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:44:27.932989 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.932929    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls\") pod \"32c4d263-879e-46b3-a4e3-7d7a313577ec\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") "
Jul 24 20:44:27.932989 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.932962    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c4d263-879e-46b3-a4e3-7d7a313577ec-openshift-service-ca\") pod \"32c4d263-879e-46b3-a4e3-7d7a313577ec\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") "
Jul 24 20:44:27.932989 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.932986    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfh2c\" (UniqueName: \"kubernetes.io/projected/32c4d263-879e-46b3-a4e3-7d7a313577ec-kube-api-access-qfh2c\") pod \"32c4d263-879e-46b3-a4e3-7d7a313577ec\" (UID: \"32c4d263-879e-46b3-a4e3-7d7a313577ec\") "
Jul 24 20:44:27.933310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.933288    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32c4d263-879e-46b3-a4e3-7d7a313577ec-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "32c4d263-879e-46b3-a4e3-7d7a313577ec" (UID: "32c4d263-879e-46b3-a4e3-7d7a313577ec"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:44:27.934872 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.934853    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32c4d263-879e-46b3-a4e3-7d7a313577ec-kube-api-access-qfh2c" (OuterVolumeSpecName: "kube-api-access-qfh2c") pod "32c4d263-879e-46b3-a4e3-7d7a313577ec" (UID: "32c4d263-879e-46b3-a4e3-7d7a313577ec"). InnerVolumeSpecName "kube-api-access-qfh2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:44:27.934958 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:27.934944    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "32c4d263-879e-46b3-a4e3-7d7a313577ec" (UID: "32c4d263-879e-46b3-a4e3-7d7a313577ec"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:44:28.034143 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.034110    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/32c4d263-879e-46b3-a4e3-7d7a313577ec-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:44:28.034143 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.034141    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/32c4d263-879e-46b3-a4e3-7d7a313577ec-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:44:28.034143 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.034151    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfh2c\" (UniqueName: \"kubernetes.io/projected/32c4d263-879e-46b3-a4e3-7d7a313577ec-kube-api-access-qfh2c\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:44:28.422946 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.422910    2566 generic.go:356] "Generic (PLEG): container finished" podID="32c4d263-879e-46b3-a4e3-7d7a313577ec" containerID="f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8" exitCode=0
Jul 24 20:44:28.423091 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.422957    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"
Jul 24 20:44:28.423091 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.422992    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm" event={"ID":"32c4d263-879e-46b3-a4e3-7d7a313577ec","Type":"ContainerDied","Data":"f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8"}
Jul 24 20:44:28.423091 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.423048    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm" event={"ID":"32c4d263-879e-46b3-a4e3-7d7a313577ec","Type":"ContainerDied","Data":"8f298f6447f37c03cf17cb559a10c1261886e321ac039d31fcd459011db0b395"}
Jul 24 20:44:28.423091 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.423070    2566 scope.go:117] "RemoveContainer" containerID="f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8"
Jul 24 20:44:28.432004 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.431985    2566 scope.go:117] "RemoveContainer" containerID="f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8"
Jul 24 20:44:28.432264 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:44:28.432247    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8\": container with ID starting with f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8 not found: ID does not exist" containerID="f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8"
Jul 24 20:44:28.432307 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.432272    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8"} err="failed to get container status \"f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8\": rpc error: code = NotFound desc = could not find container \"f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8\": container with ID starting with f9ec9dd6b332893d5d49c87dbfeb4774f6f0afe311938a0b11bdbdf35e23dcd8 not found: ID does not exist"
Jul 24 20:44:28.443501 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.443475    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"]
Jul 24 20:44:28.446629 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:28.446605    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-bd4a88b3-6c89c889df-mrctm"]
Jul 24 20:44:29.692405 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:29.692374    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="32c4d263-879e-46b3-a4e3-7d7a313577ec" path="/var/lib/kubelet/pods/32c4d263-879e-46b3-a4e3-7d7a313577ec/volumes"
Jul 24 20:44:32.333135 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:32.333078    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"]
Jul 24 20:44:32.335073 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:32.335053    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-bd4a88b3-cxsj6"]
Jul 24 20:44:33.693069 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:33.693030    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10a37fea-241e-41e5-923b-635cbb14fd77" path="/var/lib/kubelet/pods/10a37fea-241e-41e5-923b-635cbb14fd77/volumes"
Jul 24 20:44:34.456848 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.456818    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"]
Jul 24 20:44:34.457168 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457154    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy"
Jul 24 20:44:34.457168 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457168    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee8519fd-6ca3-4cfc-b996-16f764d5c638" containerName="istio-proxy"
Jul 24 20:44:34.457295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457187    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10a37fea-241e-41e5-923b-635cbb14fd77" containerName="revoke-keys"
Jul 24 20:44:34.457295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457193    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="10a37fea-241e-41e5-923b-635cbb14fd77" containerName="revoke-keys"
Jul 24 20:44:34.457295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457204    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="32c4d263-879e-46b3-a4e3-7d7a313577ec" containerName="maas-api"
Jul 24 20:44:34.457295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457210    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="32c4d263-879e-46b3-a4e3-7d7a313577ec" containerName="maas-api"
Jul 24 20:44:34.457295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457262    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="32c4d263-879e-46b3-a4e3-7d7a313577ec" containerName="maas-api"
Jul 24 20:44:34.457295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.457269    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="10a37fea-241e-41e5-923b-635cbb14fd77" containerName="revoke-keys"
Jul 24 20:44:34.461607 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.461580    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.464053 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.464038    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-2c1cd65e-openshift-default-dockercfg-5sqgb\""
Jul 24 20:44:34.471597 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.471566    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"]
Jul 24 20:44:34.485503 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485480    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9jncc\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-kube-api-access-9jncc\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-certs\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485548    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-token\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485589    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-socket\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485719 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485650    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5c8255-04f5-441c-9f13-dde3d1514621-istiod-ca-cert\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485719 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485677    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-podinfo\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485733    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-credential-socket\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485762    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-envoy\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.485864 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.485799    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-data\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.586847 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.586813    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-data\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.586855    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9jncc\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-kube-api-access-9jncc\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.586880    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-certs\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.586896    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-token\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.586918    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-socket\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587030 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.586947    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5c8255-04f5-441c-9f13-dde3d1514621-istiod-ca-cert\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587054    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-podinfo\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587161    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-credential-socket\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587194    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-envoy\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587300 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-certs\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587465 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-data\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587465 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587356    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-socket\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587565 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587509    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-credential-socket\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.587790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.587772    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5c8255-04f5-441c-9f13-dde3d1514621-istiod-ca-cert\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.589275 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.589252    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-envoy\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.589409 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.589390    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-podinfo\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.600225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.600201    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9jncc\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-kube-api-access-9jncc\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.604968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.604947    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-token\") pod \"e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") " pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.772029 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.771971    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:34.903587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.903560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"]
Jul 24 20:44:34.905515 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:44:34.905492    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d5c8255_04f5_441c_9f13_dde3d1514621.slice/crio-07e73d4504b2783868c25ef28af60d57be16904f73eed060189591337783d140 WatchSource:0}: Error finding container 07e73d4504b2783868c25ef28af60d57be16904f73eed060189591337783d140: Status 404 returned error can't find the container with id 07e73d4504b2783868c25ef28af60d57be16904f73eed060189591337783d140
Jul 24 20:44:34.907491 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.907445    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:44:34.907600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.907564    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:44:34.907664 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:34.907610    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:44:35.462059 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:35.462024    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" event={"ID":"5d5c8255-04f5-441c-9f13-dde3d1514621","Type":"ContainerStarted","Data":"437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5"}
Jul 24 20:44:35.462226 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:35.462081    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" event={"ID":"5d5c8255-04f5-441c-9f13-dde3d1514621","Type":"ContainerStarted","Data":"07e73d4504b2783868c25ef28af60d57be16904f73eed060189591337783d140"}
Jul 24 20:44:35.485147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:35.485095    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" podStartSLOduration=1.4850810939999999 podStartE2EDuration="1.485081094s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:44:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:44:35.483589777 +0000 UTC m=+1750.409728693" watchObservedRunningTime="2026-07-24 20:44:35.485081094 +0000 UTC m=+1750.411219997"
Jul 24 20:44:35.772745 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:35.772667    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:35.777120 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:35.777097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:36.467658 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:36.467628    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:36.468602 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:36.468583    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:36.906587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:36.906556    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"]
Jul 24 20:44:36.910441 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:36.910421    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:36.913124 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:36.913104    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-2c1cd65e-serving-cert\""
Jul 24 20:44:36.925015 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:36.924986    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"]
Jul 24 20:44:37.007945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.007909    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.008105 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.007966    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a5d018-09e2-4e38-8560-10b5a3764c60-openshift-service-ca\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.008105 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.008080    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnvdn\" (UniqueName: \"kubernetes.io/projected/04a5d018-09e2-4e38-8560-10b5a3764c60-kube-api-access-bnvdn\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.108945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.108907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.109127 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.108958    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a5d018-09e2-4e38-8560-10b5a3764c60-openshift-service-ca\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.109127 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.109022    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnvdn\" (UniqueName: \"kubernetes.io/projected/04a5d018-09e2-4e38-8560-10b5a3764c60-kube-api-access-bnvdn\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.109127 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:44:37.109041    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-serving-cert: secret "maas-api-e2e-derive-2c1cd65e-serving-cert" not found
Jul 24 20:44:37.109127 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:44:37.109110    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls podName:04a5d018-09e2-4e38-8560-10b5a3764c60 nodeName:}" failed. No retries permitted until 2026-07-24 20:44:37.609094993 +0000 UTC m=+1752.535233875 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls") pod "maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr" (UID: "04a5d018-09e2-4e38-8560-10b5a3764c60") : secret "maas-api-e2e-derive-2c1cd65e-serving-cert" not found
Jul 24 20:44:37.109629 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.109610    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a5d018-09e2-4e38-8560-10b5a3764c60-openshift-service-ca\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.117615 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.117593    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnvdn\" (UniqueName: \"kubernetes.io/projected/04a5d018-09e2-4e38-8560-10b5a3764c60-kube-api-access-bnvdn\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.613566 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.613508    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.616196 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.616165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls\") pod \"maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.825136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.825105    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:37.959097 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:37.959064    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"]
Jul 24 20:44:37.959812 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:44:37.959785    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04a5d018_09e2_4e38_8560_10b5a3764c60.slice/crio-bfb63d17969f5523eabf573d010d028a9da2ae54ece410c7a21627334601e22e WatchSource:0}: Error finding container bfb63d17969f5523eabf573d010d028a9da2ae54ece410c7a21627334601e22e: Status 404 returned error can't find the container with id bfb63d17969f5523eabf573d010d028a9da2ae54ece410c7a21627334601e22e
Jul 24 20:44:38.479165 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:38.479137    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr" event={"ID":"04a5d018-09e2-4e38-8560-10b5a3764c60","Type":"ContainerStarted","Data":"bfb63d17969f5523eabf573d010d028a9da2ae54ece410c7a21627334601e22e"}
Jul 24 20:44:39.485587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:39.485551    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr" event={"ID":"04a5d018-09e2-4e38-8560-10b5a3764c60","Type":"ContainerStarted","Data":"da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac"}
Jul 24 20:44:39.485957 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:39.485615    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:39.502563 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:39.502502    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr" podStartSLOduration=3.029871923 podStartE2EDuration="3.502487227s" totalImagesPullingTime="472.615304ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:44:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:44:39.500034914 +0000 UTC m=+1754.426173818" watchObservedRunningTime="2026-07-24 20:44:39.502487227 +0000 UTC m=+1754.428626127"
Jul 24 20:44:45.495933 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:45.495897    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:44:51.469172 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.469140    2566 patch_prober.go:28] interesting pod/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 24 20:44:51.469605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.469190    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 24 20:44:51.544864 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.544832    2566 generic.go:356] "Generic (PLEG): container finished" podID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerID="437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5" exitCode=0
Jul 24 20:44:51.545000 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.544905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" event={"ID":"5d5c8255-04f5-441c-9f13-dde3d1514621","Type":"ContainerDied","Data":"437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5"}
Jul 24 20:44:51.545266 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.545245    2566 scope.go:117] "RemoveContainer" containerID="437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5"
Jul 24 20:44:51.546171 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.546145    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:44:51.546267 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.546208    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:44:51.546267 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:51.546237    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:44:52.571437 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:52.571400    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" event={"ID":"5d5c8255-04f5-441c-9f13-dde3d1514621","Type":"ContainerStarted","Data":"2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc"}
Jul 24 20:44:52.773086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:52.773039    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:52.773296 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:52.773272    2566 patch_prober.go:28] interesting pod/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 24 20:44:52.773358 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:52.773337    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 24 20:44:53.772912 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:53.772880    2566 patch_prober.go:28] interesting pod/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 24 20:44:53.773308 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:53.772939    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 24 20:44:54.772745 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:54.772709    2566 patch_prober.go:28] interesting pod/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 24 20:44:54.772918 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:54.772757    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 24 20:44:55.773125 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:55.773092    2566 patch_prober.go:28] interesting pod/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused" start-of-body=
Jul 24 20:44:55.773621 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:55.773138    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.60:15021/healthz/ready\": dial tcp 10.133.0.60:15021: connect: connection refused"
Jul 24 20:44:56.776821 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:56.776791    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:56.777341 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:56.777191    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:44:56.777836 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:44:56.777813    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:45:00.127946 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.127910    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7"]
Jul 24 20:45:00.130206 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.130190    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7"
Jul 24 20:45:00.132790 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.132760    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 20:45:00.139060 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.139038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7"]
Jul 24 20:45:00.142083 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.142065    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7"
Jul 24 20:45:00.265333 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.265224    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7"]
Jul 24 20:45:00.267931 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:45:00.267903    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcda1a570_ea31_4768_80c8_b60383dc48c3.slice/crio-0fde6da7e0139c22826d807ffa7edb8283db9de7b3997ca81ce0035206247421 WatchSource:0}: Error finding container 0fde6da7e0139c22826d807ffa7edb8283db9de7b3997ca81ce0035206247421: Status 404 returned error can't find the container with id 0fde6da7e0139c22826d807ffa7edb8283db9de7b3997ca81ce0035206247421
Jul 24 20:45:00.612378 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.612350    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7" event={"ID":"cda1a570-ea31-4768-80c8-b60383dc48c3","Type":"ContainerStarted","Data":"3f8efab58536433642e39c322964eaf0427df70be8daf3064d634847bef988ac"}
Jul 24 20:45:00.612482 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.612395    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7" event={"ID":"cda1a570-ea31-4768-80c8-b60383dc48c3","Type":"ContainerStarted","Data":"0fde6da7e0139c22826d807ffa7edb8283db9de7b3997ca81ce0035206247421"}
Jul 24 20:45:00.627822 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:00.627778    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7" podStartSLOduration=0.627765514 podStartE2EDuration="627.765514ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:45:00.625782886 +0000 UTC m=+1775.551921780" watchObservedRunningTime="2026-07-24 20:45:00.627765514 +0000 UTC m=+1775.553904417"
Jul 24 20:45:01.618467 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:01.618430    2566 generic.go:356] "Generic (PLEG): container finished" podID="cda1a570-ea31-4768-80c8-b60383dc48c3" containerID="3f8efab58536433642e39c322964eaf0427df70be8daf3064d634847bef988ac" exitCode=0
Jul 24 20:45:01.618884 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:01.618520    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7" event={"ID":"cda1a570-ea31-4768-80c8-b60383dc48c3","Type":"ContainerDied","Data":"3f8efab58536433642e39c322964eaf0427df70be8daf3064d634847bef988ac"}
Jul 24 20:45:02.760412 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:02.760391    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7"
Jul 24 20:45:03.636232 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:03.636195    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7" event={"ID":"cda1a570-ea31-4768-80c8-b60383dc48c3","Type":"ContainerDied","Data":"0fde6da7e0139c22826d807ffa7edb8283db9de7b3997ca81ce0035206247421"}
Jul 24 20:45:03.636409 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:03.636242    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748765-8tgp7"
Jul 24 20:45:03.636409 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:03.636274    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fde6da7e0139c22826d807ffa7edb8283db9de7b3997ca81ce0035206247421"
Jul 24 20:45:12.572715 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:12.572673    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"]
Jul 24 20:45:12.573815 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:12.573785    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr" podUID="04a5d018-09e2-4e38-8560-10b5a3764c60" containerName="maas-api" containerID="cri-o://da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac" gracePeriod=30
Jul 24 20:45:12.832919 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:12.832867    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:45:13.014837 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.014807    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bnvdn\" (UniqueName: \"kubernetes.io/projected/04a5d018-09e2-4e38-8560-10b5a3764c60-kube-api-access-bnvdn\") pod \"04a5d018-09e2-4e38-8560-10b5a3764c60\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") "
Jul 24 20:45:13.014837 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.014841    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls\") pod \"04a5d018-09e2-4e38-8560-10b5a3764c60\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") "
Jul 24 20:45:13.015028 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.014860    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a5d018-09e2-4e38-8560-10b5a3764c60-openshift-service-ca\") pod \"04a5d018-09e2-4e38-8560-10b5a3764c60\" (UID: \"04a5d018-09e2-4e38-8560-10b5a3764c60\") "
Jul 24 20:45:13.015240 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.015214    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04a5d018-09e2-4e38-8560-10b5a3764c60-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "04a5d018-09e2-4e38-8560-10b5a3764c60" (UID: "04a5d018-09e2-4e38-8560-10b5a3764c60"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:45:13.016836 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.016810    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04a5d018-09e2-4e38-8560-10b5a3764c60-kube-api-access-bnvdn" (OuterVolumeSpecName: "kube-api-access-bnvdn") pod "04a5d018-09e2-4e38-8560-10b5a3764c60" (UID: "04a5d018-09e2-4e38-8560-10b5a3764c60"). InnerVolumeSpecName "kube-api-access-bnvdn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:45:13.016927 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.016833    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "04a5d018-09e2-4e38-8560-10b5a3764c60" (UID: "04a5d018-09e2-4e38-8560-10b5a3764c60"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:45:13.115869 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.115845    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bnvdn\" (UniqueName: \"kubernetes.io/projected/04a5d018-09e2-4e38-8560-10b5a3764c60-kube-api-access-bnvdn\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:13.115869 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.115866    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04a5d018-09e2-4e38-8560-10b5a3764c60-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:13.115998 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.115875    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04a5d018-09e2-4e38-8560-10b5a3764c60-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:13.686254 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.686216    2566 generic.go:356] "Generic (PLEG): container finished" podID="04a5d018-09e2-4e38-8560-10b5a3764c60" containerID="da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac" exitCode=0
Jul 24 20:45:13.686723 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.686325    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"
Jul 24 20:45:13.692051 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.692025    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr" event={"ID":"04a5d018-09e2-4e38-8560-10b5a3764c60","Type":"ContainerDied","Data":"da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac"}
Jul 24 20:45:13.692181 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.692077    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr" event={"ID":"04a5d018-09e2-4e38-8560-10b5a3764c60","Type":"ContainerDied","Data":"bfb63d17969f5523eabf573d010d028a9da2ae54ece410c7a21627334601e22e"}
Jul 24 20:45:13.692181 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.692110    2566 scope.go:117] "RemoveContainer" containerID="da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac"
Jul 24 20:45:13.700805 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.700789    2566 scope.go:117] "RemoveContainer" containerID="da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac"
Jul 24 20:45:13.701065 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:45:13.701026    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac\": container with ID starting with da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac not found: ID does not exist" containerID="da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac"
Jul 24 20:45:13.701105 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.701052    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac"} err="failed to get container status \"da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac\": rpc error: code = NotFound desc = could not find container \"da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac\": container with ID starting with da79b7fc91bfa942c87dd22371e2452947669e414d26e14b75cf6af7f64713ac not found: ID does not exist"
Jul 24 20:45:13.709439 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.709418    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"]
Jul 24 20:45:13.712880 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:13.712862    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-2c1cd65e-55f4ffc6f9-t7jkr"]
Jul 24 20:45:15.691948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:15.691910    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="04a5d018-09e2-4e38-8560-10b5a3764c60" path="/var/lib/kubelet/pods/04a5d018-09e2-4e38-8560-10b5a3764c60/volumes"
Jul 24 20:45:17.278656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:17.278584    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"]
Jul 24 20:45:17.279053 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:17.278863    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy" containerID="cri-o://2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc" gracePeriod=30
Jul 24 20:45:22.533494 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.533469    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:45:22.692295 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-envoy\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692317    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9jncc\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-kube-api-access-9jncc\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692343    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-data\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-token\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692385    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5c8255-04f5-441c-9f13-dde3d1514621-istiod-ca-cert\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692426    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-podinfo\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692441    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-credential-socket\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692810 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692474    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-socket\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692810 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-certs\") pod \"5d5c8255-04f5-441c-9f13-dde3d1514621\" (UID: \"5d5c8255-04f5-441c-9f13-dde3d1514621\") "
Jul 24 20:45:22.692810 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692774    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5c8255-04f5-441c-9f13-dde3d1514621-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:45:22.692954 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692854    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5c8255-04f5-441c-9f13-dde3d1514621-istiod-ca-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.692954 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.692917    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:45:22.693080 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.693054    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:45:22.693152 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.693077    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:45:22.693281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.693260    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-data" (OuterVolumeSpecName: "istio-data") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:45:22.694534 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.694482    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 20:45:22.694614 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.694593    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:45:22.694819 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.694803    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-kube-api-access-9jncc" (OuterVolumeSpecName: "kube-api-access-9jncc") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "kube-api-access-9jncc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:45:22.694858 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.694837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-token" (OuterVolumeSpecName: "istio-token") pod "5d5c8255-04f5-441c-9f13-dde3d1514621" (UID: "5d5c8255-04f5-441c-9f13-dde3d1514621"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:45:22.727641 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.727615    2566 generic.go:356] "Generic (PLEG): container finished" podID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerID="2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc" exitCode=0
Jul 24 20:45:22.727761 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.727667    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"
Jul 24 20:45:22.727761 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.727685    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" event={"ID":"5d5c8255-04f5-441c-9f13-dde3d1514621","Type":"ContainerDied","Data":"2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc"}
Jul 24 20:45:22.727761 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.727737    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz" event={"ID":"5d5c8255-04f5-441c-9f13-dde3d1514621","Type":"ContainerDied","Data":"07e73d4504b2783868c25ef28af60d57be16904f73eed060189591337783d140"}
Jul 24 20:45:22.727761 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.727758    2566 scope.go:117] "RemoveContainer" containerID="2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc"
Jul 24 20:45:22.736213 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.736197    2566 scope.go:117] "RemoveContainer" containerID="437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5"
Jul 24 20:45:22.743802 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.743784    2566 scope.go:117] "RemoveContainer" containerID="2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc"
Jul 24 20:45:22.744066 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:45:22.744049    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc\": container with ID starting with 2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc not found: ID does not exist" containerID="2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc"
Jul 24 20:45:22.744133 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.744073    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc"} err="failed to get container status \"2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc\": rpc error: code = NotFound desc = could not find container \"2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc\": container with ID starting with 2eaf3adc397f862a0af97a2dc71e673bfe916bdb4bc6bf64a822e5952eead1cc not found: ID does not exist"
Jul 24 20:45:22.744133 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.744090    2566 scope.go:117] "RemoveContainer" containerID="437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5"
Jul 24 20:45:22.744336 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:45:22.744320    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5\": container with ID starting with 437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5 not found: ID does not exist" containerID="437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5"
Jul 24 20:45:22.744384 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.744350    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5"} err="failed to get container status \"437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5\": rpc error: code = NotFound desc = could not find container \"437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5\": container with ID starting with 437fdfb47907e10df78d849cf55cc3b92fabb4349b9a530e4cb99ffee0d72ba5 not found: ID does not exist"
Jul 24 20:45:22.748404 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.748385    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"]
Jul 24 20:45:22.754580 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.754560    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-2c1cd65e-openshift-default-5b8bf76488-bnqkz"]
Jul 24 20:45:22.793420 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793399    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.793420 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793417    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-envoy\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.793520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793426    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9jncc\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-kube-api-access-9jncc\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.793520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793436    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-data\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.793520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793445    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.793520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793453    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5c8255-04f5-441c-9f13-dde3d1514621-istio-podinfo\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.793520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793460    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-credential-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:22.793520 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:22.793468    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5c8255-04f5-441c-9f13-dde3d1514621-workload-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:45:23.692885 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:45:23.692857    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" path="/var/lib/kubelet/pods/5d5c8255-04f5-441c-9f13-dde3d1514621/volumes"
Jul 24 20:46:54.648339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648254    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"]
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648829    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="04a5d018-09e2-4e38-8560-10b5a3764c60" containerName="maas-api"
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648850    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="04a5d018-09e2-4e38-8560-10b5a3764c60" containerName="maas-api"
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648864    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy"
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648873    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy"
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648882    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cda1a570-ea31-4768-80c8-b60383dc48c3" containerName="cleanup"
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648891    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="cda1a570-ea31-4768-80c8-b60383dc48c3" containerName="cleanup"
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648922    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy"
Jul 24 20:46:54.648939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.648930    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy"
Jul 24 20:46:54.649335 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.649028    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="cda1a570-ea31-4768-80c8-b60383dc48c3" containerName="cleanup"
Jul 24 20:46:54.649335 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.649041    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="04a5d018-09e2-4e38-8560-10b5a3764c60" containerName="maas-api"
Jul 24 20:46:54.649335 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.649052    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy"
Jul 24 20:46:54.649335 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.649070    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="5d5c8255-04f5-441c-9f13-dde3d1514621" containerName="istio-proxy"
Jul 24 20:46:54.651820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.651801    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.654462 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.654440    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-a3b50177-openshift-default-dockercfg-9wdfq\""
Jul 24 20:46:54.670939 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.670916    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"]
Jul 24 20:46:54.741792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.741759    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m66wl\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-kube-api-access-m66wl\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.741948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.741822    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-podinfo\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.741948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.741878    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-credential-socket\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.741948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.741901    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-certs\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.741948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.741922    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-token\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.742167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.741998    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-socket\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.742167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.742023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istiod-ca-cert\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.742167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.742060    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-envoy\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.742167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.742089    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-data\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843196 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843152    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-socket\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843455 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843425    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istiod-ca-cert\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843596 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843477    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-envoy\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843596 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843504    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-data\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843596 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843582    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m66wl\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-kube-api-access-m66wl\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843630    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-podinfo\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843681    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-credential-socket\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843693    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-socket\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843704    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-certs\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.843777 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.843726    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-token\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.844403 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.844380    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istiod-ca-cert\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.844935 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.844632    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-certs\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.844935 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.844652    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-data\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.845109 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.844811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-credential-socket\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.846673 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.846651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-envoy\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.847309 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.847290    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-podinfo\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.854502 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.854481    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-token\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.854987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.854964    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m66wl\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-kube-api-access-m66wl\") pod \"e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") " pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:54.962014 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:54.961912    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:55.105695 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:55.105663    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"]
Jul 24 20:46:55.107290 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:46:55.107267    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9ba2abba_04fb_44f3_a9be_1272ff54f58a.slice/crio-24a905eb6edd95eb713454f884ece4d76a1daf842537a116a0bf733e42c68cc5 WatchSource:0}: Error finding container 24a905eb6edd95eb713454f884ece4d76a1daf842537a116a0bf733e42c68cc5: Status 404 returned error can't find the container with id 24a905eb6edd95eb713454f884ece4d76a1daf842537a116a0bf733e42c68cc5
Jul 24 20:46:55.109331 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:55.109299    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:46:55.109406 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:55.109364    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:46:55.109406 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:55.109390    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:46:55.161319 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:55.161274    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" event={"ID":"9ba2abba-04fb-44f3-a9be-1272ff54f58a","Type":"ContainerStarted","Data":"24a905eb6edd95eb713454f884ece4d76a1daf842537a116a0bf733e42c68cc5"}
Jul 24 20:46:56.171329 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:56.171285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" event={"ID":"9ba2abba-04fb-44f3-a9be-1272ff54f58a","Type":"ContainerStarted","Data":"335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7"}
Jul 24 20:46:56.197204 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:56.197151    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" podStartSLOduration=2.197135866 podStartE2EDuration="2.197135866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:46:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:46:56.195676643 +0000 UTC m=+1891.121815567" watchObservedRunningTime="2026-07-24 20:46:56.197135866 +0000 UTC m=+1891.123274769"
Jul 24 20:46:56.963201 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:56.963159    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:56.968820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:56.968793    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:57.176592 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:57.176561    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:57.177605 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:57.177582    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:46:58.328907 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.328878    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"]
Jul 24 20:46:58.337900 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.337868    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.341248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.341223    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-a3b50177-serving-cert\""
Jul 24 20:46:58.344160 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.344125    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"]
Jul 24 20:46:58.483554 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.483506    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jqctm\" (UniqueName: \"kubernetes.io/projected/69adf26a-50e7-4328-81c6-01284d94a64b-kube-api-access-jqctm\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.483697 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.483630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69adf26a-50e7-4328-81c6-01284d94a64b-maas-api-tls\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.483760 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.483696    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69adf26a-50e7-4328-81c6-01284d94a64b-openshift-service-ca\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.584958 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.584885    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69adf26a-50e7-4328-81c6-01284d94a64b-openshift-service-ca\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.584958 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.584945    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jqctm\" (UniqueName: \"kubernetes.io/projected/69adf26a-50e7-4328-81c6-01284d94a64b-kube-api-access-jqctm\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.585139 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.585011    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69adf26a-50e7-4328-81c6-01284d94a64b-maas-api-tls\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.585675 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.585655    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69adf26a-50e7-4328-81c6-01284d94a64b-openshift-service-ca\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.587340 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.587320    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69adf26a-50e7-4328-81c6-01284d94a64b-maas-api-tls\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.595488 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.595467    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jqctm\" (UniqueName: \"kubernetes.io/projected/69adf26a-50e7-4328-81c6-01284d94a64b-kube-api-access-jqctm\") pod \"maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.652138 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.652109    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:46:58.996538 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:58.996497    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"]
Jul 24 20:46:58.998601 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:46:58.998575    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69adf26a_50e7_4328_81c6_01284d94a64b.slice/crio-2b8c4e0a09516ba911079608b49dabe902f12c040f26cfff8e07946933c41774 WatchSource:0}: Error finding container 2b8c4e0a09516ba911079608b49dabe902f12c040f26cfff8e07946933c41774: Status 404 returned error can't find the container with id 2b8c4e0a09516ba911079608b49dabe902f12c040f26cfff8e07946933c41774
Jul 24 20:46:59.188766 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:46:59.188734    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf" event={"ID":"69adf26a-50e7-4328-81c6-01284d94a64b","Type":"ContainerStarted","Data":"2b8c4e0a09516ba911079608b49dabe902f12c040f26cfff8e07946933c41774"}
Jul 24 20:47:00.194400 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:00.194361    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf" event={"ID":"69adf26a-50e7-4328-81c6-01284d94a64b","Type":"ContainerStarted","Data":"e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef"}
Jul 24 20:47:00.194868 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:00.194413    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:47:00.210863 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:00.210825    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf" podStartSLOduration=1.702788225 podStartE2EDuration="2.210810168s" totalImagesPullingTime="508.021943ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:46:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:47:00.209726951 +0000 UTC m=+1895.135865855" watchObservedRunningTime="2026-07-24 20:47:00.210810168 +0000 UTC m=+1895.136949075"
Jul 24 20:47:06.205127 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:06.205038    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:47:09.239733 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:09.239702    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerID="335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7" exitCode=0
Jul 24 20:47:09.240235 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:09.239750    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" event={"ID":"9ba2abba-04fb-44f3-a9be-1272ff54f58a","Type":"ContainerDied","Data":"335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7"}
Jul 24 20:47:09.240235 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:09.240061    2566 scope.go:117] "RemoveContainer" containerID="335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7"
Jul 24 20:47:09.240973 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:09.240946    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:47:09.241040 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:09.241012    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:47:09.241103 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:09.241043    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:47:10.251547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:10.251363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" event={"ID":"9ba2abba-04fb-44f3-a9be-1272ff54f58a","Type":"ContainerStarted","Data":"bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f"}
Jul 24 20:47:10.963072 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:10.963032    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:47:10.963899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:10.963870    2566 patch_prober.go:28] interesting pod/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 24 20:47:10.964025 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:10.963925    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 24 20:47:11.963211 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:11.963178    2566 patch_prober.go:28] interesting pod/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 24 20:47:11.963694 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:11.963234    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 24 20:47:12.177173 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:12.177115    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:47:12.963572 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:12.963245    2566 patch_prober.go:28] interesting pod/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused" start-of-body=
Jul 24 20:47:12.963572 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:12.963312    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.63:15021/healthz/ready\": dial tcp 10.133.0.63:15021: connect: connection refused"
Jul 24 20:47:13.966157 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:13.966129    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:47:13.967031 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:13.967012    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:47:20.488488 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.488450    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"]
Jul 24 20:47:20.488934 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.488671    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf" podUID="69adf26a-50e7-4328-81c6-01284d94a64b" containerName="maas-api" containerID="cri-o://e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef" gracePeriod=30
Jul 24 20:47:20.738230 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.738204    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:47:20.886297 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.886268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jqctm\" (UniqueName: \"kubernetes.io/projected/69adf26a-50e7-4328-81c6-01284d94a64b-kube-api-access-jqctm\") pod \"69adf26a-50e7-4328-81c6-01284d94a64b\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") "
Jul 24 20:47:20.886461 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.886304    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69adf26a-50e7-4328-81c6-01284d94a64b-maas-api-tls\") pod \"69adf26a-50e7-4328-81c6-01284d94a64b\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") "
Jul 24 20:47:20.886461 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.886436    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69adf26a-50e7-4328-81c6-01284d94a64b-openshift-service-ca\") pod \"69adf26a-50e7-4328-81c6-01284d94a64b\" (UID: \"69adf26a-50e7-4328-81c6-01284d94a64b\") "
Jul 24 20:47:20.886795 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.886769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69adf26a-50e7-4328-81c6-01284d94a64b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "69adf26a-50e7-4328-81c6-01284d94a64b" (UID: "69adf26a-50e7-4328-81c6-01284d94a64b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:47:20.888275 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.888254    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69adf26a-50e7-4328-81c6-01284d94a64b-kube-api-access-jqctm" (OuterVolumeSpecName: "kube-api-access-jqctm") pod "69adf26a-50e7-4328-81c6-01284d94a64b" (UID: "69adf26a-50e7-4328-81c6-01284d94a64b"). InnerVolumeSpecName "kube-api-access-jqctm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:47:20.888339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.888258    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/69adf26a-50e7-4328-81c6-01284d94a64b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "69adf26a-50e7-4328-81c6-01284d94a64b" (UID: "69adf26a-50e7-4328-81c6-01284d94a64b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:47:20.988077 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.988050    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/69adf26a-50e7-4328-81c6-01284d94a64b-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:20.988077 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.988075    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jqctm\" (UniqueName: \"kubernetes.io/projected/69adf26a-50e7-4328-81c6-01284d94a64b-kube-api-access-jqctm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:20.988237 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:20.988085    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/69adf26a-50e7-4328-81c6-01284d94a64b-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:21.305722 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.305641    2566 generic.go:356] "Generic (PLEG): container finished" podID="69adf26a-50e7-4328-81c6-01284d94a64b" containerID="e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef" exitCode=0
Jul 24 20:47:21.305722 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.305698    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"
Jul 24 20:47:21.305935 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.305726    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf" event={"ID":"69adf26a-50e7-4328-81c6-01284d94a64b","Type":"ContainerDied","Data":"e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef"}
Jul 24 20:47:21.305935 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.305778    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf" event={"ID":"69adf26a-50e7-4328-81c6-01284d94a64b","Type":"ContainerDied","Data":"2b8c4e0a09516ba911079608b49dabe902f12c040f26cfff8e07946933c41774"}
Jul 24 20:47:21.305935 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.305801    2566 scope.go:117] "RemoveContainer" containerID="e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef"
Jul 24 20:47:21.314421 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.314405    2566 scope.go:117] "RemoveContainer" containerID="e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef"
Jul 24 20:47:21.314679 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:47:21.314660    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef\": container with ID starting with e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef not found: ID does not exist" containerID="e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef"
Jul 24 20:47:21.314735 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.314687    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef"} err="failed to get container status \"e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef\": rpc error: code = NotFound desc = could not find container \"e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef\": container with ID starting with e50317a774835916aacf1f74e8437e2396c91cddd8425ec2c60b84eebb74c0ef not found: ID does not exist"
Jul 24 20:47:21.325730 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.325706    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"]
Jul 24 20:47:21.328086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.328068    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-a3b50177-65c98f8cf8-bq8jf"]
Jul 24 20:47:21.692254 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:21.692223    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69adf26a-50e7-4328-81c6-01284d94a64b" path="/var/lib/kubelet/pods/69adf26a-50e7-4328-81c6-01284d94a64b/volumes"
Jul 24 20:47:31.679474 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:31.679439    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"]
Jul 24 20:47:31.679922 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:31.679721    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy" containerID="cri-o://bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f" gracePeriod=30
Jul 24 20:47:36.924194 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:36.924167    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:47:37.007808 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.007736    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-envoy\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.007808 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.007777    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-socket\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.007816    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-credential-socket\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.007850    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-token\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.007873    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-certs\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.007904    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-data\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.007944    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istiod-ca-cert\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008009 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008006    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m66wl\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-kube-api-access-m66wl\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008312 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008030    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-podinfo\") pod \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\" (UID: \"9ba2abba-04fb-44f3-a9be-1272ff54f58a\") "
Jul 24 20:47:37.008312 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008127    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:47:37.008312 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008273    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:47:37.008552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008323    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:47:37.008552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008358    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:47:37.008552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008449    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.008552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008471    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-credential-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.008552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008490    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-workload-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.008552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.008504    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istiod-ca-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.009112 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.009078    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-data" (OuterVolumeSpecName: "istio-data") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:47:37.010242 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.010217    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 20:47:37.010242 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.010225    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-kube-api-access-m66wl" (OuterVolumeSpecName: "kube-api-access-m66wl") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "kube-api-access-m66wl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:47:37.010242 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.010220    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:47:37.010413 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.010283    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-token" (OuterVolumeSpecName: "istio-token") pod "9ba2abba-04fb-44f3-a9be-1272ff54f58a" (UID: "9ba2abba-04fb-44f3-a9be-1272ff54f58a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:47:37.109301 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.109269    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m66wl\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-kube-api-access-m66wl\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.109301 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.109294    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-podinfo\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.109301 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.109304    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-envoy\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.109511 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.109313    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.109511 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.109320    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9ba2abba-04fb-44f3-a9be-1272ff54f58a-istio-data\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:47:37.381662 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.381630    2566 generic.go:356] "Generic (PLEG): container finished" podID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerID="bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f" exitCode=0
Jul 24 20:47:37.381829 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.381685    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"
Jul 24 20:47:37.381829 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.381702    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" event={"ID":"9ba2abba-04fb-44f3-a9be-1272ff54f58a","Type":"ContainerDied","Data":"bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f"}
Jul 24 20:47:37.381829 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.381759    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g" event={"ID":"9ba2abba-04fb-44f3-a9be-1272ff54f58a","Type":"ContainerDied","Data":"24a905eb6edd95eb713454f884ece4d76a1daf842537a116a0bf733e42c68cc5"}
Jul 24 20:47:37.381829 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.381782    2566 scope.go:117] "RemoveContainer" containerID="bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f"
Jul 24 20:47:37.390837 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.390815    2566 scope.go:117] "RemoveContainer" containerID="335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7"
Jul 24 20:47:37.399106 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.399090    2566 scope.go:117] "RemoveContainer" containerID="bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f"
Jul 24 20:47:37.399393 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:47:37.399373    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f\": container with ID starting with bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f not found: ID does not exist" containerID="bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f"
Jul 24 20:47:37.399439 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.399401    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f"} err="failed to get container status \"bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f\": rpc error: code = NotFound desc = could not find container \"bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f\": container with ID starting with bcf1e7024309554e1f0ea63a01089103ff8f94df9cdc2681baeaf1152462175f not found: ID does not exist"
Jul 24 20:47:37.399439 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.399418    2566 scope.go:117] "RemoveContainer" containerID="335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7"
Jul 24 20:47:37.399683 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:47:37.399665    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7\": container with ID starting with 335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7 not found: ID does not exist" containerID="335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7"
Jul 24 20:47:37.399739 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.399689    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7"} err="failed to get container status \"335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7\": rpc error: code = NotFound desc = could not find container \"335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7\": container with ID starting with 335595f2923273dcc3702dc4fca8fc26d567889bdfc337fac59afe38e85a2ed7 not found: ID does not exist"
Jul 24 20:47:37.403681 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.403660    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"]
Jul 24 20:47:37.406995 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.406976    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-a3b50177-openshift-default-85f4754c6d-sp46g"]
Jul 24 20:47:37.692261 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:37.692186    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" path="/var/lib/kubelet/pods/9ba2abba-04fb-44f3-a9be-1272ff54f58a/volumes"
Jul 24 20:47:54.293567 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.293505    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"]
Jul 24 20:47:54.294041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.293905    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy"
Jul 24 20:47:54.294041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.293916    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy"
Jul 24 20:47:54.294041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.293941    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69adf26a-50e7-4328-81c6-01284d94a64b" containerName="maas-api"
Jul 24 20:47:54.294041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.293946    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="69adf26a-50e7-4328-81c6-01284d94a64b" containerName="maas-api"
Jul 24 20:47:54.294041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.294010    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy"
Jul 24 20:47:54.294041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.294018    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy"
Jul 24 20:47:54.294041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.294023    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="69adf26a-50e7-4328-81c6-01284d94a64b" containerName="maas-api"
Jul 24 20:47:54.298716 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.298688    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.301489 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.301469    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-760f3539-serving-cert\""
Jul 24 20:47:54.310045 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.310018    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"]
Jul 24 20:47:54.360057 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.360020    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-llp75\" (UniqueName: \"kubernetes.io/projected/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-kube-api-access-llp75\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.360232 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.360125    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-openshift-service-ca\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.360232 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.360154    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.460758 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.460720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-openshift-service-ca\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.460948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.460774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.460948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.460847    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-llp75\" (UniqueName: \"kubernetes.io/projected/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-kube-api-access-llp75\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.460948 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:47:54.460884    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-serving-cert: secret "maas-api-e2e-mt-760f3539-serving-cert" not found
Jul 24 20:47:54.461074 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:47:54.460964    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls podName:fcc86b19-d6ec-46b3-bbe7-d82db21b61dc nodeName:}" failed. No retries permitted until 2026-07-24 20:47:54.960947265 +0000 UTC m=+1949.887086147 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls") pod "maas-api-e2e-mt-760f3539-648587f4d5-gk8h5" (UID: "fcc86b19-d6ec-46b3-bbe7-d82db21b61dc") : secret "maas-api-e2e-mt-760f3539-serving-cert" not found
Jul 24 20:47:54.461401 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.461379    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-openshift-service-ca\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.468998 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.468969    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-llp75\" (UniqueName: \"kubernetes.io/projected/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-kube-api-access-llp75\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.872108 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.872074    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"]
Jul 24 20:47:54.872433 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.872421    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy"
Jul 24 20:47:54.872481 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.872435    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="9ba2abba-04fb-44f3-a9be-1272ff54f58a" containerName="istio-proxy"
Jul 24 20:47:54.875744 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.875728    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:54.878549 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.878499    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-760f3539-dockercfg-cl4lb\""
Jul 24 20:47:54.878649 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.878552    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-760f3539\""
Jul 24 20:47:54.884024 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.884001    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"]
Jul 24 20:47:54.964615 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.964579    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a253225-2593-42fd-99fb-7792335309fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:54.964805 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.964635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qppfw\" (UniqueName: \"kubernetes.io/projected/2a253225-2593-42fd-99fb-7792335309fb-kube-api-access-qppfw\") pod \"payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:54.964805 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.964745    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:54.967090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:54.967065    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls\") pod \"maas-api-e2e-mt-760f3539-648587f4d5-gk8h5\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:55.065332 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.065300    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a253225-2593-42fd-99fb-7792335309fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:55.065508 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.065355    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qppfw\" (UniqueName: \"kubernetes.io/projected/2a253225-2593-42fd-99fb-7792335309fb-kube-api-access-qppfw\") pod \"payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:55.065998 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.065972    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a253225-2593-42fd-99fb-7792335309fb-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:55.074493 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.074463    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qppfw\" (UniqueName: \"kubernetes.io/projected/2a253225-2593-42fd-99fb-7792335309fb-kube-api-access-qppfw\") pod \"payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:55.188899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.188803    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:55.210182 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.210040    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:55.343624 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.343560    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"]
Jul 24 20:47:55.347401 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:47:55.347366    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a253225_2593_42fd_99fb_7792335309fb.slice/crio-3bf4357a9ea14160947b8dd8b7e92e00b374a7f2279d941445a64a5dbd65fce3 WatchSource:0}: Error finding container 3bf4357a9ea14160947b8dd8b7e92e00b374a7f2279d941445a64a5dbd65fce3: Status 404 returned error can't find the container with id 3bf4357a9ea14160947b8dd8b7e92e00b374a7f2279d941445a64a5dbd65fce3
Jul 24 20:47:55.366715 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.366689    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"]
Jul 24 20:47:55.369674 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:47:55.369647    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcc86b19_d6ec_46b3_bbe7_d82db21b61dc.slice/crio-33c14634016d16b04b78cdaf9018021f7f524c29d19a383a30fbe8a14f364c54 WatchSource:0}: Error finding container 33c14634016d16b04b78cdaf9018021f7f524c29d19a383a30fbe8a14f364c54: Status 404 returned error can't find the container with id 33c14634016d16b04b78cdaf9018021f7f524c29d19a383a30fbe8a14f364c54
Jul 24 20:47:55.370812 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.370794    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 20:47:55.470735 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.470646    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5" event={"ID":"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc","Type":"ContainerStarted","Data":"33c14634016d16b04b78cdaf9018021f7f524c29d19a383a30fbe8a14f364c54"}
Jul 24 20:47:55.472340 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.472314    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj" event={"ID":"2a253225-2593-42fd-99fb-7792335309fb","Type":"ContainerStarted","Data":"53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6"}
Jul 24 20:47:55.472486 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.472359    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj" event={"ID":"2a253225-2593-42fd-99fb-7792335309fb","Type":"ContainerStarted","Data":"3bf4357a9ea14160947b8dd8b7e92e00b374a7f2279d941445a64a5dbd65fce3"}
Jul 24 20:47:55.472486 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.472452    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:47:55.489146 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:55.489093    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj" podStartSLOduration=1.489072693 podStartE2EDuration="1.489072693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:47:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:47:55.487025148 +0000 UTC m=+1950.413164053" watchObservedRunningTime="2026-07-24 20:47:55.489072693 +0000 UTC m=+1950.415211595"
Jul 24 20:47:56.479401 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:56.479362    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5" event={"ID":"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc","Type":"ContainerStarted","Data":"4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59"}
Jul 24 20:47:56.479868 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:56.479609    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:47:56.496875 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:47:56.496823    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5" podStartSLOduration=2.088080623 podStartE2EDuration="2.496805349s" totalImagesPullingTime="408.724726ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:47:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:47:56.495126079 +0000 UTC m=+1951.421264987" watchObservedRunningTime="2026-07-24 20:47:56.496805349 +0000 UTC m=+1951.422944252"
Jul 24 20:48:02.489584 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:02.489558    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:48:06.482125 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:06.482097    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:48:28.260607 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.260569    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"]
Jul 24 20:48:28.265656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.265634    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"
Jul 24 20:48:28.268402 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.268380    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 20:48:28.271308 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.271282    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"]
Jul 24 20:48:28.332676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.332653    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b402d1-8c08-4600-bdec-79cbeab2d8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-760f3539-k7984\" (UID: \"e7b402d1-8c08-4600-bdec-79cbeab2d8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"
Jul 24 20:48:28.433298 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.433272    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b402d1-8c08-4600-bdec-79cbeab2d8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-760f3539-k7984\" (UID: \"e7b402d1-8c08-4600-bdec-79cbeab2d8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"
Jul 24 20:48:28.433845 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.433823    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b402d1-8c08-4600-bdec-79cbeab2d8cb-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-760f3539-k7984\" (UID: \"e7b402d1-8c08-4600-bdec-79cbeab2d8cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"
Jul 24 20:48:28.577988 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.577933    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"
Jul 24 20:48:28.709047 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:28.709015    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"]
Jul 24 20:48:28.712097 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:48:28.712059    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7b402d1_8c08_4600_bdec_79cbeab2d8cb.slice/crio-40b5c7a75cd295f5443f0da8b7aead3b2b65c2c16e87dad26b44d5959ad304a5 WatchSource:0}: Error finding container 40b5c7a75cd295f5443f0da8b7aead3b2b65c2c16e87dad26b44d5959ad304a5: Status 404 returned error can't find the container with id 40b5c7a75cd295f5443f0da8b7aead3b2b65c2c16e87dad26b44d5959ad304a5
Jul 24 20:48:29.637776 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:29.637689    2566 generic.go:356] "Generic (PLEG): container finished" podID="e7b402d1-8c08-4600-bdec-79cbeab2d8cb" containerID="4a64187edf99160baaf01506bf5582012e6b53d0f6464bc6c7c344d071e28a75" exitCode=0
Jul 24 20:48:29.638197 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:29.637768    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984" event={"ID":"e7b402d1-8c08-4600-bdec-79cbeab2d8cb","Type":"ContainerDied","Data":"4a64187edf99160baaf01506bf5582012e6b53d0f6464bc6c7c344d071e28a75"}
Jul 24 20:48:29.638197 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:29.637820    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984" event={"ID":"e7b402d1-8c08-4600-bdec-79cbeab2d8cb","Type":"ContainerStarted","Data":"40b5c7a75cd295f5443f0da8b7aead3b2b65c2c16e87dad26b44d5959ad304a5"}
Jul 24 20:48:30.773922 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:30.773903    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"
Jul 24 20:48:30.853618 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:30.853583    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b402d1-8c08-4600-bdec-79cbeab2d8cb-maas-api-service-ca\") pod \"e7b402d1-8c08-4600-bdec-79cbeab2d8cb\" (UID: \"e7b402d1-8c08-4600-bdec-79cbeab2d8cb\") "
Jul 24 20:48:30.853929 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:30.853904    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7b402d1-8c08-4600-bdec-79cbeab2d8cb-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e7b402d1-8c08-4600-bdec-79cbeab2d8cb" (UID: "e7b402d1-8c08-4600-bdec-79cbeab2d8cb"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:48:30.954167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:30.954085    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e7b402d1-8c08-4600-bdec-79cbeab2d8cb-maas-api-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:48:31.649809 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:31.649772    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984" event={"ID":"e7b402d1-8c08-4600-bdec-79cbeab2d8cb","Type":"ContainerDied","Data":"40b5c7a75cd295f5443f0da8b7aead3b2b65c2c16e87dad26b44d5959ad304a5"}
Jul 24 20:48:31.649809 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:31.649813    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"
Jul 24 20:48:31.650010 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:31.649862    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="40b5c7a75cd295f5443f0da8b7aead3b2b65c2c16e87dad26b44d5959ad304a5"
Jul 24 20:48:43.677163 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:43.677129    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"]
Jul 24 20:48:43.677698 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:43.677350    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5" podUID="fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" containerName="maas-api" containerID="cri-o://4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59" gracePeriod=30
Jul 24 20:48:43.735362 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:43.735334    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"]
Jul 24 20:48:43.735569 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:43.735516    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj" podUID="2a253225-2593-42fd-99fb-7792335309fb" containerName="payload-pre-processing" containerID="cri-o://53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6" gracePeriod=30
Jul 24 20:48:44.017356 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.017330    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:48:44.019979 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.019960    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:48:44.067719 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.067688    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-openshift-service-ca\") pod \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") "
Jul 24 20:48:44.067866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.067737    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a253225-2593-42fd-99fb-7792335309fb-plugins-config-volume\") pod \"2a253225-2593-42fd-99fb-7792335309fb\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") "
Jul 24 20:48:44.067866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.067760    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-llp75\" (UniqueName: \"kubernetes.io/projected/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-kube-api-access-llp75\") pod \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") "
Jul 24 20:48:44.067866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.067788    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls\") pod \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\" (UID: \"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc\") "
Jul 24 20:48:44.067866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.067827    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qppfw\" (UniqueName: \"kubernetes.io/projected/2a253225-2593-42fd-99fb-7792335309fb-kube-api-access-qppfw\") pod \"2a253225-2593-42fd-99fb-7792335309fb\" (UID: \"2a253225-2593-42fd-99fb-7792335309fb\") "
Jul 24 20:48:44.068095 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.068063    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" (UID: "fcc86b19-d6ec-46b3-bbe7-d82db21b61dc"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:48:44.068158 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.068104    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a253225-2593-42fd-99fb-7792335309fb-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2a253225-2593-42fd-99fb-7792335309fb" (UID: "2a253225-2593-42fd-99fb-7792335309fb"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:48:44.068216 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.068167    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:48:44.069860 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.069837    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a253225-2593-42fd-99fb-7792335309fb-kube-api-access-qppfw" (OuterVolumeSpecName: "kube-api-access-qppfw") pod "2a253225-2593-42fd-99fb-7792335309fb" (UID: "2a253225-2593-42fd-99fb-7792335309fb"). InnerVolumeSpecName "kube-api-access-qppfw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:48:44.070235 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.070215    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" (UID: "fcc86b19-d6ec-46b3-bbe7-d82db21b61dc"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:48:44.070235 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.070226    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-kube-api-access-llp75" (OuterVolumeSpecName: "kube-api-access-llp75") pod "fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" (UID: "fcc86b19-d6ec-46b3-bbe7-d82db21b61dc"). InnerVolumeSpecName "kube-api-access-llp75". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:48:44.169433 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.169409    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:48:44.169568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.169436    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qppfw\" (UniqueName: \"kubernetes.io/projected/2a253225-2593-42fd-99fb-7792335309fb-kube-api-access-qppfw\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:48:44.169568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.169450    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2a253225-2593-42fd-99fb-7792335309fb-plugins-config-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:48:44.169568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.169464    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-llp75\" (UniqueName: \"kubernetes.io/projected/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc-kube-api-access-llp75\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:48:44.709291 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.709259    2566 generic.go:356] "Generic (PLEG): container finished" podID="fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" containerID="4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59" exitCode=0
Jul 24 20:48:44.709780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.709308    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"
Jul 24 20:48:44.709780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.709353    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5" event={"ID":"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc","Type":"ContainerDied","Data":"4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59"}
Jul 24 20:48:44.709780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.709411    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5" event={"ID":"fcc86b19-d6ec-46b3-bbe7-d82db21b61dc","Type":"ContainerDied","Data":"33c14634016d16b04b78cdaf9018021f7f524c29d19a383a30fbe8a14f364c54"}
Jul 24 20:48:44.709780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.709446    2566 scope.go:117] "RemoveContainer" containerID="4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59"
Jul 24 20:48:44.710792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.710618    2566 generic.go:356] "Generic (PLEG): container finished" podID="2a253225-2593-42fd-99fb-7792335309fb" containerID="53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6" exitCode=0
Jul 24 20:48:44.710792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.710658    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj" event={"ID":"2a253225-2593-42fd-99fb-7792335309fb","Type":"ContainerDied","Data":"53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6"}
Jul 24 20:48:44.710792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.710688    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"
Jul 24 20:48:44.710792 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.710692    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj" event={"ID":"2a253225-2593-42fd-99fb-7792335309fb","Type":"ContainerDied","Data":"3bf4357a9ea14160947b8dd8b7e92e00b374a7f2279d941445a64a5dbd65fce3"}
Jul 24 20:48:44.718573 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.718558    2566 scope.go:117] "RemoveContainer" containerID="4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59"
Jul 24 20:48:44.718832 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:48:44.718813    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59\": container with ID starting with 4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59 not found: ID does not exist" containerID="4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59"
Jul 24 20:48:44.718890 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.718838    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59"} err="failed to get container status \"4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59\": rpc error: code = NotFound desc = could not find container \"4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59\": container with ID starting with 4bbe7cd4fd3955cd02dc566b82325f2bf889fae02bd14308e782aa95bbb35d59 not found: ID does not exist"
Jul 24 20:48:44.718890 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.718854    2566 scope.go:117] "RemoveContainer" containerID="53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6"
Jul 24 20:48:44.728179 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.728162    2566 scope.go:117] "RemoveContainer" containerID="53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6"
Jul 24 20:48:44.728440 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:48:44.728424    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6\": container with ID starting with 53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6 not found: ID does not exist" containerID="53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6"
Jul 24 20:48:44.728501 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.728444    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6"} err="failed to get container status \"53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6\": rpc error: code = NotFound desc = could not find container \"53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6\": container with ID starting with 53a59a23396cb7c3ae719c4bbed7d195f2ffe902f45f6d1cd746d4e1020e84a6 not found: ID does not exist"
Jul 24 20:48:44.734258 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.734228    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"]
Jul 24 20:48:44.736684 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.736660    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-760f3539-648587f4d5-gk8h5"]
Jul 24 20:48:44.747678 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.747658    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"]
Jul 24 20:48:44.751587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:44.751569    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-760f3539-6f5dbd7648-64hcj"]
Jul 24 20:48:45.692727 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:45.692686    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2a253225-2593-42fd-99fb-7792335309fb" path="/var/lib/kubelet/pods/2a253225-2593-42fd-99fb-7792335309fb/volumes"
Jul 24 20:48:45.693164 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:45.693142    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" path="/var/lib/kubelet/pods/fcc86b19-d6ec-46b3-bbe7-d82db21b61dc/volumes"
Jul 24 20:48:48.330230 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:48.330195    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"]
Jul 24 20:48:48.332704 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:48.332681    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-760f3539-k7984"]
Jul 24 20:48:49.692453 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:48:49.692410    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e7b402d1-8c08-4600-bdec-79cbeab2d8cb" path="/var/lib/kubelet/pods/e7b402d1-8c08-4600-bdec-79cbeab2d8cb/volumes"
Jul 24 20:49:25.733987 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:49:25.733956    2566 scope.go:117] "RemoveContainer" containerID="11cb9dfa67278fa33bb6c7ca84c743df72690091466c2bc1e19df4878c497d36"
Jul 24 20:50:25.757443 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:25.757411    2566 scope.go:117] "RemoveContainer" containerID="102628e874817f46a11d74e8dd0e02b6fbb37c407ace357be54f5fededf0c57e"
Jul 24 20:50:41.864642 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.864602    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"]
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.864968    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" containerName="maas-api"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.864980    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" containerName="maas-api"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.864989    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e7b402d1-8c08-4600-bdec-79cbeab2d8cb" containerName="revoke-keys"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.864995    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="e7b402d1-8c08-4600-bdec-79cbeab2d8cb" containerName="revoke-keys"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.865003    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2a253225-2593-42fd-99fb-7792335309fb" containerName="payload-pre-processing"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.865009    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="2a253225-2593-42fd-99fb-7792335309fb" containerName="payload-pre-processing"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.865080    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fcc86b19-d6ec-46b3-bbe7-d82db21b61dc" containerName="maas-api"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.865098    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="e7b402d1-8c08-4600-bdec-79cbeab2d8cb" containerName="revoke-keys"
Jul 24 20:50:41.865616 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.865109    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="2a253225-2593-42fd-99fb-7792335309fb" containerName="payload-pre-processing"
Jul 24 20:50:41.868238 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.868219    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.870820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.870799    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-c97e06-openshift-default-dockercfg-vvsx2\""
Jul 24 20:50:41.878410 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.878385    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"]
Jul 24 20:50:41.891344 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891323    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-socket\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891451 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-envoy\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891451 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891390    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-data\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891451 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891437    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxvtw\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-kube-api-access-bxvtw\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891633 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891463    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-credential-socket\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891633 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891502    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e40be5-384b-453e-85f2-aaf1b284c592-istiod-ca-cert\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891633 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891588    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-istio-token\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891645    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-certs\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.891786 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.891690    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e40be5-384b-453e-85f2-aaf1b284c592-istio-podinfo\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992482 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992452    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-envoy\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992482 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992485    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-data\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992516    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxvtw\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-kube-api-access-bxvtw\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992567    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-credential-socket\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e40be5-384b-453e-85f2-aaf1b284c592-istiod-ca-cert\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992644    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-istio-token\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-certs\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.992724 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992723    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e40be5-384b-453e-85f2-aaf1b284c592-istio-podinfo\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.993042 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992793    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-socket\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.993042 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992929    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-data\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.993042 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.992995    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-credential-socket\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.993206 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.993187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-certs\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.993252 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.993189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-socket\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.993316 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.993296    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e40be5-384b-453e-85f2-aaf1b284c592-istiod-ca-cert\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.994799 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.994783    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-envoy\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:41.995120 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:41.995103    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e40be5-384b-453e-85f2-aaf1b284c592-istio-podinfo\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:42.002164 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:42.002144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-istio-token\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:42.002663 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:42.002642    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxvtw\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-kube-api-access-bxvtw\") pod \"e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") " pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:42.178748 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:42.178673    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:42.309730 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:42.309698    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"]
Jul 24 20:50:42.311053 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:50:42.311018    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47e40be5_384b_453e_85f2_aaf1b284c592.slice/crio-0872e55b9cc6ebb3d1cdb36455b33a70db32f9145cba201c041da795403c9ec9 WatchSource:0}: Error finding container 0872e55b9cc6ebb3d1cdb36455b33a70db32f9145cba201c041da795403c9ec9: Status 404 returned error can't find the container with id 0872e55b9cc6ebb3d1cdb36455b33a70db32f9145cba201c041da795403c9ec9
Jul 24 20:50:42.313471 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:42.313436    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:50:42.313574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:42.313502    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:50:42.313574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:42.313569    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:50:43.250285 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:43.250247    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" event={"ID":"47e40be5-384b-453e-85f2-aaf1b284c592","Type":"ContainerStarted","Data":"dee0b955d21144b72cf74daee3d7ef06c414f6312c9b83f41492fef5392aa7d9"}
Jul 24 20:50:43.250704 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:43.250308    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" event={"ID":"47e40be5-384b-453e-85f2-aaf1b284c592","Type":"ContainerStarted","Data":"0872e55b9cc6ebb3d1cdb36455b33a70db32f9145cba201c041da795403c9ec9"}
Jul 24 20:50:43.271573 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:43.271515    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" podStartSLOduration=2.271501364 podStartE2EDuration="2.271501364s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:50:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:50:43.269535087 +0000 UTC m=+2118.195674001" watchObservedRunningTime="2026-07-24 20:50:43.271501364 +0000 UTC m=+2118.197640267"
Jul 24 20:50:44.179391 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:44.179354    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:44.183771 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:44.183745    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:44.255253 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:44.255225    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:44.256037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:44.256020    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:45.414277 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.414249    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"]
Jul 24 20:50:45.418174 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.418145    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.420869 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.420844    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-c97e06-serving-cert\""
Jul 24 20:50:45.432418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.432378    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"]
Jul 24 20:50:45.525750 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.525718    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.525907 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.525763    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6hf9\" (UniqueName: \"kubernetes.io/projected/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-kube-api-access-x6hf9\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.525907 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.525836    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.626750 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.626720    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.626899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.626764    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6hf9\" (UniqueName: \"kubernetes.io/projected/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-kube-api-access-x6hf9\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.626899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.626804    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.626899 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:50:45.626881    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-serving-cert: secret "maas-api-e2e-inf-a-c97e06-serving-cert" not found
Jul 24 20:50:45.627015 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:50:45.626994    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls podName:92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98 nodeName:}" failed. No retries permitted until 2026-07-24 20:50:46.126972105 +0000 UTC m=+2121.053111004 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls") pod "maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p" (UID: "92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98") : secret "maas-api-e2e-inf-a-c97e06-serving-cert" not found
Jul 24 20:50:45.627352 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.627336    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-openshift-service-ca\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:45.638691 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:45.638661    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6hf9\" (UniqueName: \"kubernetes.io/projected/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-kube-api-access-x6hf9\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:46.132283 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:46.132241    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:46.134643 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:46.134617    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls\") pod \"maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:46.331379 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:46.331350    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:46.671836 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:46.671811    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"]
Jul 24 20:50:46.673747 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:50:46.673717    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92cc2eb7_b8e1_4ca2_8f69_985d25a0fc98.slice/crio-4ccf02e770da204229440b3e72d00904347036c2a6a6b6529c7e2b5714567282 WatchSource:0}: Error finding container 4ccf02e770da204229440b3e72d00904347036c2a6a6b6529c7e2b5714567282: Status 404 returned error can't find the container with id 4ccf02e770da204229440b3e72d00904347036c2a6a6b6529c7e2b5714567282
Jul 24 20:50:47.269959 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:47.269925    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p" event={"ID":"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98","Type":"ContainerStarted","Data":"4ccf02e770da204229440b3e72d00904347036c2a6a6b6529c7e2b5714567282"}
Jul 24 20:50:48.277245 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:48.277203    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p" event={"ID":"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98","Type":"ContainerStarted","Data":"bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1"}
Jul 24 20:50:48.277696 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:48.277333    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:48.294575 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:48.294512    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p" podStartSLOduration=2.688529909 podStartE2EDuration="3.294497843s" totalImagesPullingTime="605.967934ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:50:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:50:48.292862974 +0000 UTC m=+2123.219001899" watchObservedRunningTime="2026-07-24 20:50:48.294497843 +0000 UTC m=+2123.220636797"
Jul 24 20:50:53.279933 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.279890    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"]
Jul 24 20:50:53.287810 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.287786    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.290482 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.290458    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-55d9b1-serving-cert\""
Jul 24 20:50:53.299329 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.299307    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"]
Jul 24 20:50:53.397396 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.397365    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-maas-api-tls\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.397584 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.397440    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dllx\" (UniqueName: \"kubernetes.io/projected/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-kube-api-access-2dllx\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.397584 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.397485    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.498065 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.498028    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-maas-api-tls\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.498260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.498122    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dllx\" (UniqueName: \"kubernetes.io/projected/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-kube-api-access-2dllx\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.498260 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.498164    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.498828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.498807    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-openshift-service-ca\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.500393 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.500378    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-maas-api-tls\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.506512 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.506489    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dllx\" (UniqueName: \"kubernetes.io/projected/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-kube-api-access-2dllx\") pod \"maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.601677 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.601649    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:53.764552 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:53.763113    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"]
Jul 24 20:50:54.116422 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.116394    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"]
Jul 24 20:50:54.123717 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.123695    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.126406 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.126374    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-55d9b1\""
Jul 24 20:50:54.126534 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.126411    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-55d9b1-dockercfg-5cvc2\""
Jul 24 20:50:54.129763 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.129743    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"]
Jul 24 20:50:54.205291 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.205264    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8x929\" (UniqueName: \"kubernetes.io/projected/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-kube-api-access-8x929\") pod \"payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.205420 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.205355    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.288666 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.288643    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:50:54.306210 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.306183    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8x929\" (UniqueName: \"kubernetes.io/projected/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-kube-api-access-8x929\") pod \"payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.306347 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.306283    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.307408 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.307377    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.312114 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.312090    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g" event={"ID":"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf","Type":"ContainerStarted","Data":"cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af"}
Jul 24 20:50:54.312205 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.312140    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g" event={"ID":"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf","Type":"ContainerStarted","Data":"dd40b1589b55bd2618ad4b5ab9150b8ff4e587356e881fb6495c7d0c23628c4e"}
Jul 24 20:50:54.312205 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.312176    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:50:54.314912 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.314893    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8x929\" (UniqueName: \"kubernetes.io/projected/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-kube-api-access-8x929\") pod \"payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") " pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.330979 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.330928    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g" podStartSLOduration=0.910833678 podStartE2EDuration="1.330910968s" totalImagesPullingTime="420.07729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:50:54.32962164 +0000 UTC m=+2129.255760545" watchObservedRunningTime="2026-07-24 20:50:54.330910968 +0000 UTC m=+2129.257049873"
Jul 24 20:50:54.435365 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.435289    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:54.630954 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:54.630599    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"]
Jul 24 20:50:54.633368 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:50:54.633340    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb5df3b43_f772_4842_ad9c_4c1d3aa15e85.slice/crio-f4699c1f0edc638e000c06fa7afb6307358fed74979abcce98bea3dc3db55825 WatchSource:0}: Error finding container f4699c1f0edc638e000c06fa7afb6307358fed74979abcce98bea3dc3db55825: Status 404 returned error can't find the container with id f4699c1f0edc638e000c06fa7afb6307358fed74979abcce98bea3dc3db55825
Jul 24 20:50:55.318980 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:55.318937    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n" event={"ID":"b5df3b43-f772-4842-ad9c-4c1d3aa15e85","Type":"ContainerStarted","Data":"052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c"}
Jul 24 20:50:55.319408 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:55.318995    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n" event={"ID":"b5df3b43-f772-4842-ad9c-4c1d3aa15e85","Type":"ContainerStarted","Data":"f4699c1f0edc638e000c06fa7afb6307358fed74979abcce98bea3dc3db55825"}
Jul 24 20:50:55.319408 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:55.319052    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:50:55.336401 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:55.336341    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n" podStartSLOduration=1.336323016 podStartE2EDuration="1.336323016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:50:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:50:55.335436707 +0000 UTC m=+2130.261575612" watchObservedRunningTime="2026-07-24 20:50:55.336323016 +0000 UTC m=+2130.262461921"
Jul 24 20:50:58.334883 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:58.334855    2566 generic.go:356] "Generic (PLEG): container finished" podID="47e40be5-384b-453e-85f2-aaf1b284c592" containerID="dee0b955d21144b72cf74daee3d7ef06c414f6312c9b83f41492fef5392aa7d9" exitCode=0
Jul 24 20:50:58.335197 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:58.334899    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" event={"ID":"47e40be5-384b-453e-85f2-aaf1b284c592","Type":"ContainerDied","Data":"dee0b955d21144b72cf74daee3d7ef06c414f6312c9b83f41492fef5392aa7d9"}
Jul 24 20:50:58.335248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:58.335234    2566 scope.go:117] "RemoveContainer" containerID="dee0b955d21144b72cf74daee3d7ef06c414f6312c9b83f41492fef5392aa7d9"
Jul 24 20:50:58.336384 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:58.336352    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:50:58.336452 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:58.336436    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:50:58.336487 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:58.336475    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:50:59.255646 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:59.255591    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:50:59.344204 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:50:59.344162    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" event={"ID":"47e40be5-384b-453e-85f2-aaf1b284c592","Type":"ContainerStarted","Data":"51a987bb950e5bc39f993b4fde4dbd37ec99518340cf0a7a2efe3f691f6d86a1"}
Jul 24 20:51:00.179409 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:00.179374    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:51:00.180510 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:00.180490    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 20:51:00.180590 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:00.180554    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 20:51:00.324600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:00.324574    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:51:01.179958 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:01.179920    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 20:51:01.180428 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:01.179984    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 20:51:02.179880 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:02.179848    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused" start-of-body=
Jul 24 20:51:02.180045 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:02.179900    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.68:15021/healthz/ready\": dial tcp 10.133.0.68:15021: connect: connection refused"
Jul 24 20:51:03.182721 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:03.182690    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:51:03.183183 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:03.183068    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:51:03.183746 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:03.183729    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:51:06.327086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:06.327055    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:51:24.083120 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.083045    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"]
Jul 24 20:51:24.087259 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.087238    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.090168 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.090059    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-55d9b1\"/\"openshift-service-ca.crt\""
Jul 24 20:51:24.091298 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.091280    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-55d9b1\"/\"test-model-55d9b1-kserve-self-signed-certs\""
Jul 24 20:51:24.091298 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.091289    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-55d9b1\"/\"kube-root-ca.crt\""
Jul 24 20:51:24.091472 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.091328    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-55d9b1\"/\"default-dockercfg-7jdn2\""
Jul 24 20:51:24.095059 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.095038    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"]
Jul 24 20:51:24.260302 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.260270    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-home\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.260302 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.260304    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8mndb\" (UniqueName: \"kubernetes.io/projected/8351ab80-c160-4aa5-b2fe-b5d16f535020-kube-api-access-8mndb\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.260580 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.260351    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-tmp-dir\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.260580 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.260386    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-dshm\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.260580 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.260401    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8351ab80-c160-4aa5-b2fe-b5d16f535020-tls-certs\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.260580 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.260473    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-model-cache\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.361676 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.361647    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-home\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.361828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.361679    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8mndb\" (UniqueName: \"kubernetes.io/projected/8351ab80-c160-4aa5-b2fe-b5d16f535020-kube-api-access-8mndb\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.361828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.361719    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-tmp-dir\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.361828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.361740    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-dshm\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.361828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.361755    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8351ab80-c160-4aa5-b2fe-b5d16f535020-tls-certs\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.361828 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.361784    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-model-cache\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.362071 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.362011    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-home\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.362126 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.362105    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-tmp-dir\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.362206 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.362189    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-model-cache\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.364110 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.364088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-dshm\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.364226 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.364210    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8351ab80-c160-4aa5-b2fe-b5d16f535020-tls-certs\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.370079 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.370061    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8mndb\" (UniqueName: \"kubernetes.io/projected/8351ab80-c160-4aa5-b2fe-b5d16f535020-kube-api-access-8mndb\") pod \"test-model-55d9b1-kserve-58987ffd45-s898k\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") " pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.398072 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.398055    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:24.734019 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:24.733996    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"]
Jul 24 20:51:24.735689 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:51:24.735656    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8351ab80_c160_4aa5_b2fe_b5d16f535020.slice/crio-e98891f7c7181569038b2c1f3ef9906f77a0b716dcc740574ae22b50fbf70af1 WatchSource:0}: Error finding container e98891f7c7181569038b2c1f3ef9906f77a0b716dcc740574ae22b50fbf70af1: Status 404 returned error can't find the container with id e98891f7c7181569038b2c1f3ef9906f77a0b716dcc740574ae22b50fbf70af1
Jul 24 20:51:25.482925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:25.482896    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k" event={"ID":"8351ab80-c160-4aa5-b2fe-b5d16f535020","Type":"ContainerStarted","Data":"f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802"}
Jul 24 20:51:25.483283 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:25.482932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k" event={"ID":"8351ab80-c160-4aa5-b2fe-b5d16f535020","Type":"ContainerStarted","Data":"e98891f7c7181569038b2c1f3ef9906f77a0b716dcc740574ae22b50fbf70af1"}
Jul 24 20:51:25.500925 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:25.500879    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k" podStartSLOduration=1.500866157 podStartE2EDuration="1.500866157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:51:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:51:25.49864171 +0000 UTC m=+2160.424780627" watchObservedRunningTime="2026-07-24 20:51:25.500866157 +0000 UTC m=+2160.427005060"
Jul 24 20:51:34.398270 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:34.398235    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:34.398727 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:34.398285    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:34.410633 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:34.410604    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:51:34.533355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:51:34.533322    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:52:08.171414 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:08.171381    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"]
Jul 24 20:52:18.656088 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.656050    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"]
Jul 24 20:52:18.656548 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.656320    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p" podUID="92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" containerName="maas-api" containerID="cri-o://bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1" gracePeriod=30
Jul 24 20:52:18.910429 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.910373    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:52:18.921167 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.921143    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6hf9\" (UniqueName: \"kubernetes.io/projected/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-kube-api-access-x6hf9\") pod \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") "
Jul 24 20:52:18.921264 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.921192    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-openshift-service-ca\") pod \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") "
Jul 24 20:52:18.921264 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.921218    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls\") pod \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\" (UID: \"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98\") "
Jul 24 20:52:18.921502 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.921477    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" (UID: "92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:52:18.923020 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.923000    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-kube-api-access-x6hf9" (OuterVolumeSpecName: "kube-api-access-x6hf9") pod "92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" (UID: "92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98"). InnerVolumeSpecName "kube-api-access-x6hf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:52:18.923191 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:18.923171    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" (UID: "92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:52:19.021820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.021792    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6hf9\" (UniqueName: \"kubernetes.io/projected/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-kube-api-access-x6hf9\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:19.021820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.021818    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:19.021968 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.021829    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:19.736400 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.736367    2566 generic.go:356] "Generic (PLEG): container finished" podID="92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" containerID="bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1" exitCode=0
Jul 24 20:52:19.736780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.736417    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"
Jul 24 20:52:19.736780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.736449    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p" event={"ID":"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98","Type":"ContainerDied","Data":"bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1"}
Jul 24 20:52:19.736780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.736508    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p" event={"ID":"92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98","Type":"ContainerDied","Data":"4ccf02e770da204229440b3e72d00904347036c2a6a6b6529c7e2b5714567282"}
Jul 24 20:52:19.736780 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.736554    2566 scope.go:117] "RemoveContainer" containerID="bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1"
Jul 24 20:52:19.747016 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.746999    2566 scope.go:117] "RemoveContainer" containerID="bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1"
Jul 24 20:52:19.747317 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:52:19.747292    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1\": container with ID starting with bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1 not found: ID does not exist" containerID="bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1"
Jul 24 20:52:19.747391 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.747327    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1"} err="failed to get container status \"bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1\": rpc error: code = NotFound desc = could not find container \"bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1\": container with ID starting with bd1d4c24b881b2a35956d384bf92f77c295493ffb54b0f87dabfda32998287c1 not found: ID does not exist"
Jul 24 20:52:19.760971 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.760162    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"]
Jul 24 20:52:19.760971 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:19.760207    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-c97e06-6c488c6b8-vjn9p"]
Jul 24 20:52:21.692390 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:21.692359    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" path="/var/lib/kubelet/pods/92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98/volumes"
Jul 24 20:52:23.189948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.189912    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k" podUID="8351ab80-c160-4aa5-b2fe-b5d16f535020" containerName="main" containerID="cri-o://f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802" gracePeriod=45
Jul 24 20:52:23.441778 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.441706    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:52:23.456839 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.456818    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-dshm\") pod \"8351ab80-c160-4aa5-b2fe-b5d16f535020\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") "
Jul 24 20:52:23.456955 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.456852    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8351ab80-c160-4aa5-b2fe-b5d16f535020-tls-certs\") pod \"8351ab80-c160-4aa5-b2fe-b5d16f535020\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") "
Jul 24 20:52:23.456955 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.456885    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-tmp-dir\") pod \"8351ab80-c160-4aa5-b2fe-b5d16f535020\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") "
Jul 24 20:52:23.456955 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.456909    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-home\") pod \"8351ab80-c160-4aa5-b2fe-b5d16f535020\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") "
Jul 24 20:52:23.457105 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.457034    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8mndb\" (UniqueName: \"kubernetes.io/projected/8351ab80-c160-4aa5-b2fe-b5d16f535020-kube-api-access-8mndb\") pod \"8351ab80-c160-4aa5-b2fe-b5d16f535020\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") "
Jul 24 20:52:23.457165 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.457121    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-model-cache\") pod \"8351ab80-c160-4aa5-b2fe-b5d16f535020\" (UID: \"8351ab80-c160-4aa5-b2fe-b5d16f535020\") "
Jul 24 20:52:23.457215 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.457160    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-home" (OuterVolumeSpecName: "home") pod "8351ab80-c160-4aa5-b2fe-b5d16f535020" (UID: "8351ab80-c160-4aa5-b2fe-b5d16f535020"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:23.457215 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.457173    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8351ab80-c160-4aa5-b2fe-b5d16f535020" (UID: "8351ab80-c160-4aa5-b2fe-b5d16f535020"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:23.457437 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.457419    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-tmp-dir\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:23.457437 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.457441    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-home\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:23.457437 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.457459    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-model-cache" (OuterVolumeSpecName: "model-cache") pod "8351ab80-c160-4aa5-b2fe-b5d16f535020" (UID: "8351ab80-c160-4aa5-b2fe-b5d16f535020"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:23.459767 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.459669    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-dshm" (OuterVolumeSpecName: "dshm") pod "8351ab80-c160-4aa5-b2fe-b5d16f535020" (UID: "8351ab80-c160-4aa5-b2fe-b5d16f535020"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:23.460228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.460203    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8351ab80-c160-4aa5-b2fe-b5d16f535020-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8351ab80-c160-4aa5-b2fe-b5d16f535020" (UID: "8351ab80-c160-4aa5-b2fe-b5d16f535020"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:52:23.460459 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.460436    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8351ab80-c160-4aa5-b2fe-b5d16f535020-kube-api-access-8mndb" (OuterVolumeSpecName: "kube-api-access-8mndb") pod "8351ab80-c160-4aa5-b2fe-b5d16f535020" (UID: "8351ab80-c160-4aa5-b2fe-b5d16f535020"). InnerVolumeSpecName "kube-api-access-8mndb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:52:23.558290 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.558188    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8mndb\" (UniqueName: \"kubernetes.io/projected/8351ab80-c160-4aa5-b2fe-b5d16f535020-kube-api-access-8mndb\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:23.558290 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.558225    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-model-cache\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:23.558290 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.558240    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8351ab80-c160-4aa5-b2fe-b5d16f535020-dshm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:23.558290 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.558251    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8351ab80-c160-4aa5-b2fe-b5d16f535020-tls-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:23.758175 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.758144    2566 generic.go:356] "Generic (PLEG): container finished" podID="8351ab80-c160-4aa5-b2fe-b5d16f535020" containerID="f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802" exitCode=0
Jul 24 20:52:23.758175 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.758176    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k" event={"ID":"8351ab80-c160-4aa5-b2fe-b5d16f535020","Type":"ContainerDied","Data":"f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802"}
Jul 24 20:52:23.758361 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.758182    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"
Jul 24 20:52:23.758361 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.758200    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k" event={"ID":"8351ab80-c160-4aa5-b2fe-b5d16f535020","Type":"ContainerDied","Data":"e98891f7c7181569038b2c1f3ef9906f77a0b716dcc740574ae22b50fbf70af1"}
Jul 24 20:52:23.758361 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.758216    2566 scope.go:117] "RemoveContainer" containerID="f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802"
Jul 24 20:52:23.766647 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.766630    2566 scope.go:117] "RemoveContainer" containerID="f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802"
Jul 24 20:52:23.766909 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:52:23.766875    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802\": container with ID starting with f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802 not found: ID does not exist" containerID="f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802"
Jul 24 20:52:23.767013 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.766917    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802"} err="failed to get container status \"f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802\": rpc error: code = NotFound desc = could not find container \"f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802\": container with ID starting with f0d16c84a885a5bb5cfb29835e48619b8c3d55990caac95ef717874ec895d802 not found: ID does not exist"
Jul 24 20:52:23.774415 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.774395    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"]
Jul 24 20:52:23.778008 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:23.777989    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-55d9b1/test-model-55d9b1-kserve-58987ffd45-s898k"]
Jul 24 20:52:25.692547 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:25.692500    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8351ab80-c160-4aa5-b2fe-b5d16f535020" path="/var/lib/kubelet/pods/8351ab80-c160-4aa5-b2fe-b5d16f535020/volumes"
Jul 24 20:52:29.681480 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:29.681440    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"]
Jul 24 20:52:29.681958 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:29.681694    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy" containerID="cri-o://51a987bb950e5bc39f993b4fde4dbd37ec99518340cf0a7a2efe3f691f6d86a1" gracePeriod=30
Jul 24 20:52:33.185626 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:33.185593    2566 patch_prober.go:28] interesting pod/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 20:52:33.185991 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:33.185652    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 20:52:34.817066 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:34.817019    2566 generic.go:356] "Generic (PLEG): container finished" podID="47e40be5-384b-453e-85f2-aaf1b284c592" containerID="51a987bb950e5bc39f993b4fde4dbd37ec99518340cf0a7a2efe3f691f6d86a1" exitCode=0
Jul 24 20:52:34.817484 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:34.817342    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" event={"ID":"47e40be5-384b-453e-85f2-aaf1b284c592","Type":"ContainerDied","Data":"51a987bb950e5bc39f993b4fde4dbd37ec99518340cf0a7a2efe3f691f6d86a1"}
Jul 24 20:52:34.817484 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:34.817418    2566 scope.go:117] "RemoveContainer" containerID="dee0b955d21144b72cf74daee3d7ef06c414f6312c9b83f41492fef5392aa7d9"
Jul 24 20:52:34.944235 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:34.944154    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:52:35.059050 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059018    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-istio-token\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059070    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-data\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059093    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e40be5-384b-453e-85f2-aaf1b284c592-istiod-ca-cert\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059116    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-certs\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059132    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e40be5-384b-453e-85f2-aaf1b284c592-istio-podinfo\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059160    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-socket\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059228 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059191    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-envoy\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059564 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059244    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-credential-socket\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059564 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059307    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bxvtw\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-kube-api-access-bxvtw\") pod \"47e40be5-384b-453e-85f2-aaf1b284c592\" (UID: \"47e40be5-384b-453e-85f2-aaf1b284c592\") "
Jul 24 20:52:35.059564 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059502    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47e40be5-384b-453e-85f2-aaf1b284c592-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:52:35.059744 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059635    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/47e40be5-384b-453e-85f2-aaf1b284c592-istiod-ca-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.059744 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059644    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:35.059857 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059771    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:35.059857 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.059818    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:35.060027 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.060005    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-data" (OuterVolumeSpecName: "istio-data") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:35.061418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.061392    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-istio-token" (OuterVolumeSpecName: "istio-token") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:52:35.061418 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.061409    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/47e40be5-384b-453e-85f2-aaf1b284c592-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 20:52:35.061594 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.061415    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:52:35.061594 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.061521    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-kube-api-access-bxvtw" (OuterVolumeSpecName: "kube-api-access-bxvtw") pod "47e40be5-384b-453e-85f2-aaf1b284c592" (UID: "47e40be5-384b-453e-85f2-aaf1b284c592"). InnerVolumeSpecName "kube-api-access-bxvtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:52:35.160136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160099    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxvtw\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-kube-api-access-bxvtw\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.160136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160130    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/47e40be5-384b-453e-85f2-aaf1b284c592-istio-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.160136 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160141    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-data\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.160376 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160151    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.160376 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160159    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/47e40be5-384b-453e-85f2-aaf1b284c592-istio-podinfo\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.160376 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160166    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-workload-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.160376 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160175    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-istio-envoy\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.160376 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.160184    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/47e40be5-384b-453e-85f2-aaf1b284c592-credential-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:35.823437 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.823407    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw" event={"ID":"47e40be5-384b-453e-85f2-aaf1b284c592","Type":"ContainerDied","Data":"0872e55b9cc6ebb3d1cdb36455b33a70db32f9145cba201c041da795403c9ec9"}
Jul 24 20:52:35.823437 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.823427    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"
Jul 24 20:52:35.823962 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.823465    2566 scope.go:117] "RemoveContainer" containerID="51a987bb950e5bc39f993b4fde4dbd37ec99518340cf0a7a2efe3f691f6d86a1"
Jul 24 20:52:35.849115 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.849092    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"]
Jul 24 20:52:35.858745 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:35.858726    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-c97e06-openshift-default-57c7f87986-nthjw"]
Jul 24 20:52:37.692203 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:37.692167    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" path="/var/lib/kubelet/pods/47e40be5-384b-453e-85f2-aaf1b284c592/volumes"
Jul 24 20:52:40.436219 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.436186    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"]
Jul 24 20:52:40.437038 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.436977    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g" podUID="0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" containerName="maas-api" containerID="cri-o://cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af" gracePeriod=30
Jul 24 20:52:40.483905 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.483878    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"]
Jul 24 20:52:40.484121 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.484071    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n" podUID="b5df3b43-f772-4842-ad9c-4c1d3aa15e85" containerName="payload-processing" containerID="cri-o://052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c" gracePeriod=30
Jul 24 20:52:40.732031 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.732009    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:52:40.761538 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.761503    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:52:40.817518 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.817485    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2dllx\" (UniqueName: \"kubernetes.io/projected/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-kube-api-access-2dllx\") pod \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") "
Jul 24 20:52:40.817666 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.817521    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8x929\" (UniqueName: \"kubernetes.io/projected/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-kube-api-access-8x929\") pod \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") "
Jul 24 20:52:40.817666 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.817570    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-openshift-service-ca\") pod \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") "
Jul 24 20:52:40.817804 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.817667    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-maas-api-tls\") pod \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\" (UID: \"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf\") "
Jul 24 20:52:40.817804 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.817729    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-plugins-config-volume\") pod \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\" (UID: \"b5df3b43-f772-4842-ad9c-4c1d3aa15e85\") "
Jul 24 20:52:40.817990 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.817968    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" (UID: "0ab104b0-8bc4-41ba-9c63-174fb6c01ddf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:52:40.818091 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.818069    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b5df3b43-f772-4842-ad9c-4c1d3aa15e85" (UID: "b5df3b43-f772-4842-ad9c-4c1d3aa15e85"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:52:40.819645 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.819623    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-kube-api-access-8x929" (OuterVolumeSpecName: "kube-api-access-8x929") pod "b5df3b43-f772-4842-ad9c-4c1d3aa15e85" (UID: "b5df3b43-f772-4842-ad9c-4c1d3aa15e85"). InnerVolumeSpecName "kube-api-access-8x929". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:52:40.819728 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.819651    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-kube-api-access-2dllx" (OuterVolumeSpecName: "kube-api-access-2dllx") pod "0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" (UID: "0ab104b0-8bc4-41ba-9c63-174fb6c01ddf"). InnerVolumeSpecName "kube-api-access-2dllx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:52:40.819728 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.819670    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" (UID: "0ab104b0-8bc4-41ba-9c63-174fb6c01ddf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:52:40.849588 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.849557    2566 generic.go:356] "Generic (PLEG): container finished" podID="b5df3b43-f772-4842-ad9c-4c1d3aa15e85" containerID="052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c" exitCode=0
Jul 24 20:52:40.849687 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.849591    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"
Jul 24 20:52:40.849687 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.849627    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n" event={"ID":"b5df3b43-f772-4842-ad9c-4c1d3aa15e85","Type":"ContainerDied","Data":"052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c"}
Jul 24 20:52:40.849769 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.849706    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n" event={"ID":"b5df3b43-f772-4842-ad9c-4c1d3aa15e85","Type":"ContainerDied","Data":"f4699c1f0edc638e000c06fa7afb6307358fed74979abcce98bea3dc3db55825"}
Jul 24 20:52:40.849769 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.849730    2566 scope.go:117] "RemoveContainer" containerID="052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c"
Jul 24 20:52:40.851069 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.851014    2566 generic.go:356] "Generic (PLEG): container finished" podID="0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" containerID="cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af" exitCode=0
Jul 24 20:52:40.851069 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.851042    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g" event={"ID":"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf","Type":"ContainerDied","Data":"cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af"}
Jul 24 20:52:40.851303 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.851089    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g" event={"ID":"0ab104b0-8bc4-41ba-9c63-174fb6c01ddf","Type":"ContainerDied","Data":"dd40b1589b55bd2618ad4b5ab9150b8ff4e587356e881fb6495c7d0c23628c4e"}
Jul 24 20:52:40.851303 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.851051    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"
Jul 24 20:52:40.859128 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.859113    2566 scope.go:117] "RemoveContainer" containerID="052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c"
Jul 24 20:52:40.859464 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:52:40.859377    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c\": container with ID starting with 052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c not found: ID does not exist" containerID="052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c"
Jul 24 20:52:40.859464 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.859410    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c"} err="failed to get container status \"052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c\": rpc error: code = NotFound desc = could not find container \"052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c\": container with ID starting with 052e3d6ebf12d90533ff52baf2ec6a931436eeec68929f0727e3b71d9316b53c not found: ID does not exist"
Jul 24 20:52:40.859464 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.859433    2566 scope.go:117] "RemoveContainer" containerID="cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af"
Jul 24 20:52:40.866983 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.866967    2566 scope.go:117] "RemoveContainer" containerID="cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af"
Jul 24 20:52:40.867206 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:52:40.867188    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af\": container with ID starting with cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af not found: ID does not exist" containerID="cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af"
Jul 24 20:52:40.867254 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.867212    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af"} err="failed to get container status \"cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af\": rpc error: code = NotFound desc = could not find container \"cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af\": container with ID starting with cc8929c3d53b2971096e2f0a5aa0b1a37ff9c9ebd09d5f41b53c6d68650107af not found: ID does not exist"
Jul 24 20:52:40.874344 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.874324    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"]
Jul 24 20:52:40.879334 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.879316    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-55d9b1-6b589f9557-k277g"]
Jul 24 20:52:40.888670 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.888649    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"]
Jul 24 20:52:40.891033 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.891013    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-55d9b1-8447fd9d48-lsf2n"]
Jul 24 20:52:40.918969 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.918946    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:40.918969 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.918967    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-plugins-config-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:40.919094 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.918982    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dllx\" (UniqueName: \"kubernetes.io/projected/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-kube-api-access-2dllx\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:40.919094 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.918995    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8x929\" (UniqueName: \"kubernetes.io/projected/b5df3b43-f772-4842-ad9c-4c1d3aa15e85-kube-api-access-8x929\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:40.919094 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:40.919007    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:52:41.693334 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:41.693306    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" path="/var/lib/kubelet/pods/0ab104b0-8bc4-41ba-9c63-174fb6c01ddf/volumes"
Jul 24 20:52:41.693705 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:41.693666    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b5df3b43-f772-4842-ad9c-4c1d3aa15e85" path="/var/lib/kubelet/pods/b5df3b43-f772-4842-ad9c-4c1d3aa15e85/volumes"
Jul 24 20:52:52.727436 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727348    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"]
Jul 24 20:52:52.727897 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727878    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy"
Jul 24 20:52:52.727945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727901    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy"
Jul 24 20:52:52.727945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727919    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" containerName="maas-api"
Jul 24 20:52:52.727945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727928    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" containerName="maas-api"
Jul 24 20:52:52.727945 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727938    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8351ab80-c160-4aa5-b2fe-b5d16f535020" containerName="main"
Jul 24 20:52:52.728090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727946    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="8351ab80-c160-4aa5-b2fe-b5d16f535020" containerName="main"
Jul 24 20:52:52.728090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727974    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" containerName="maas-api"
Jul 24 20:52:52.728090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.727984    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" containerName="maas-api"
Jul 24 20:52:52.728090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728002    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b5df3b43-f772-4842-ad9c-4c1d3aa15e85" containerName="payload-processing"
Jul 24 20:52:52.728090 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728010    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="b5df3b43-f772-4842-ad9c-4c1d3aa15e85" containerName="payload-processing"
Jul 24 20:52:52.728269 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728097    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy"
Jul 24 20:52:52.728269 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728109    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy"
Jul 24 20:52:52.728269 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728119    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="92cc2eb7-b8e1-4ca2-8f69-985d25a0fc98" containerName="maas-api"
Jul 24 20:52:52.728269 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728126    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="b5df3b43-f772-4842-ad9c-4c1d3aa15e85" containerName="payload-processing"
Jul 24 20:52:52.728269 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728136    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ab104b0-8bc4-41ba-9c63-174fb6c01ddf" containerName="maas-api"
Jul 24 20:52:52.728269 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.728143    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="8351ab80-c160-4aa5-b2fe-b5d16f535020" containerName="main"
Jul 24 20:52:52.732600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.732583    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.735734 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.735700    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-678134-openshift-default-dockercfg-zfpml\""
Jul 24 20:52:52.743641 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.743618    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"]
Jul 24 20:52:52.823665 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823630    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9423d64-6531-4919-b315-1af882795022-istiod-ca-cert\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.823872 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823766    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-envoy\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.823872 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823797    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-istio-token\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.823872 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823850    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-socket\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.824046 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823872    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t48zp\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-kube-api-access-t48zp\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.824046 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823921    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9423d64-6531-4919-b315-1af882795022-istio-podinfo\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.824046 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823949    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-credential-socket\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.824046 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.823988    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-certs\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.824218 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.824107    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-data\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925302 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9423d64-6531-4919-b315-1af882795022-istio-podinfo\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925302 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925301    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-credential-socket\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925319    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-certs\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925447    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-data\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925511    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9423d64-6531-4919-b315-1af882795022-istiod-ca-cert\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925767 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925622    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-envoy\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925767 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925650    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-istio-token\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925767 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925700    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-socket\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925767 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t48zp\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-kube-api-access-t48zp\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.925767 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925734    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-certs\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.926021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925804    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-credential-socket\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.926021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.925853    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-data\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.926119 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.926038    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-socket\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.926271 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.926251    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9423d64-6531-4919-b315-1af882795022-istiod-ca-cert\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.927757 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.927739    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-envoy\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.927819 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.927778    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9423d64-6531-4919-b315-1af882795022-istio-podinfo\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.933656 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.933633    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-istio-token\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:52.934308 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:52.934289    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t48zp\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-kube-api-access-t48zp\") pod \"e2e-api-a-678134-openshift-default-b757565f8-v5j82\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") " pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:53.046453 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.046365    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:53.184380 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.184356    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"]
Jul 24 20:52:53.185936 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:52:53.185902    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9423d64_6531_4919_b315_1af882795022.slice/crio-9827257b3f90aa6684bdbd3424a95e5e83c38339bea9c21f047e7215149cbc52 WatchSource:0}: Error finding container 9827257b3f90aa6684bdbd3424a95e5e83c38339bea9c21f047e7215149cbc52: Status 404 returned error can't find the container with id 9827257b3f90aa6684bdbd3424a95e5e83c38339bea9c21f047e7215149cbc52
Jul 24 20:52:53.187930 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.187899    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:52:53.188053 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.187964    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:52:53.188053 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.187994    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:52:53.917807 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.917773    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" event={"ID":"d9423d64-6531-4919-b315-1af882795022","Type":"ContainerStarted","Data":"7be09376a1d25a2bf85e78ed6b58e69e7524007aff07f82e6d9dd69e656b18ea"}
Jul 24 20:52:53.918201 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.917834    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" event={"ID":"d9423d64-6531-4919-b315-1af882795022","Type":"ContainerStarted","Data":"9827257b3f90aa6684bdbd3424a95e5e83c38339bea9c21f047e7215149cbc52"}
Jul 24 20:52:53.940424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:53.940369    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podStartSLOduration=1.940352067 podStartE2EDuration="1.940352067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:52:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:52:53.937970273 +0000 UTC m=+2248.864109177" watchObservedRunningTime="2026-07-24 20:52:53.940352067 +0000 UTC m=+2248.866490970"
Jul 24 20:52:54.046948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:54.046913    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:54.055401 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:54.055373    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:54.922170 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:54.922139    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:54.923212 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:54.923187    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:52:55.988732 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:55.988691    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"]
Jul 24 20:52:55.989189 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:55.989049    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy"
Jul 24 20:52:55.989189 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:55.989059    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="47e40be5-384b-453e-85f2-aaf1b284c592" containerName="istio-proxy"
Jul 24 20:52:55.992119 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:55.992100    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:55.994759 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:55.994738    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-678134-serving-cert\""
Jul 24 20:52:56.008162 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.008129    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"]
Jul 24 20:52:56.156483 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.156446    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8mjf\" (UniqueName: \"kubernetes.io/projected/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-kube-api-access-c8mjf\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.156725 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.156513    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-maas-api-tls\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.156725 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.156635    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-openshift-service-ca\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.258004 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.257907    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-maas-api-tls\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.258004 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.257974    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-openshift-service-ca\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.258223 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.258008    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8mjf\" (UniqueName: \"kubernetes.io/projected/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-kube-api-access-c8mjf\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.258712 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.258687    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-openshift-service-ca\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.260336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.260317    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-maas-api-tls\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.265975 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.265955    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8mjf\" (UniqueName: \"kubernetes.io/projected/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-kube-api-access-c8mjf\") pod \"maas-api-e2e-api-a-678134-7547b9d496-8fctb\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.305548 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.305499    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:56.445971 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.445939    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"]
Jul 24 20:52:56.446684 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:52:56.446658    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod266a5e2b_f5ad_496a_aaab_61b5bcb9481a.slice/crio-fa57cf16774b409115ef83b5ec57a659733d53c812a00fb46151e601809e4bac WatchSource:0}: Error finding container fa57cf16774b409115ef83b5ec57a659733d53c812a00fb46151e601809e4bac: Status 404 returned error can't find the container with id fa57cf16774b409115ef83b5ec57a659733d53c812a00fb46151e601809e4bac
Jul 24 20:52:56.448754 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.448732    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 20:52:56.932782 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:56.932738    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb" event={"ID":"266a5e2b-f5ad-496a-aaab-61b5bcb9481a","Type":"ContainerStarted","Data":"fa57cf16774b409115ef83b5ec57a659733d53c812a00fb46151e601809e4bac"}
Jul 24 20:52:57.939164 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:57.939119    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb" event={"ID":"266a5e2b-f5ad-496a-aaab-61b5bcb9481a","Type":"ContainerStarted","Data":"b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f"}
Jul 24 20:52:57.939589 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:57.939238    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:52:57.958065 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:52:57.958020    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb" podStartSLOduration=2.492474606 podStartE2EDuration="2.958005652s" totalImagesPullingTime="465.531046ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:52:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:52:57.956082918 +0000 UTC m=+2252.882221821" watchObservedRunningTime="2026-07-24 20:52:57.958005652 +0000 UTC m=+2252.884144576"
Jul 24 20:53:03.949285 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:03.949256    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:53:04.022191 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.022158    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"]
Jul 24 20:53:04.025851 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.025828    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.028411 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.028385    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-1ae1a9-serving-cert\""
Jul 24 20:53:04.037611 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.037553    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"]
Jul 24 20:53:04.116126 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.116092    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfcfb2-238e-4e0e-9b00-9609f0356212-openshift-service-ca\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.116313 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.116138    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfcfb2-238e-4e0e-9b00-9609f0356212-maas-api-tls\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.116313 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.116296    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxzb5\" (UniqueName: \"kubernetes.io/projected/fadfcfb2-238e-4e0e-9b00-9609f0356212-kube-api-access-xxzb5\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.217587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.217487    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxzb5\" (UniqueName: \"kubernetes.io/projected/fadfcfb2-238e-4e0e-9b00-9609f0356212-kube-api-access-xxzb5\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.217587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.217576    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfcfb2-238e-4e0e-9b00-9609f0356212-openshift-service-ca\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.217763 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.217609    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfcfb2-238e-4e0e-9b00-9609f0356212-maas-api-tls\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.218204 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.218186    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfcfb2-238e-4e0e-9b00-9609f0356212-openshift-service-ca\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.219960 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.219937    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfcfb2-238e-4e0e-9b00-9609f0356212-maas-api-tls\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.228623 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.228601    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxzb5\" (UniqueName: \"kubernetes.io/projected/fadfcfb2-238e-4e0e-9b00-9609f0356212-kube-api-access-xxzb5\") pod \"maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.338598 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.338555    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.478383 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.478336    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"]
Jul 24 20:53:04.481272 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:53:04.481244    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfadfcfb2_238e_4e0e_9b00_9609f0356212.slice/crio-7386b2a292dcbe35b9d89c9fdbe9500de107e3493f2ff4a8236c583be7db7654 WatchSource:0}: Error finding container 7386b2a292dcbe35b9d89c9fdbe9500de107e3493f2ff4a8236c583be7db7654: Status 404 returned error can't find the container with id 7386b2a292dcbe35b9d89c9fdbe9500de107e3493f2ff4a8236c583be7db7654
Jul 24 20:53:04.974669 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.974632    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2" event={"ID":"fadfcfb2-238e-4e0e-9b00-9609f0356212","Type":"ContainerStarted","Data":"198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae"}
Jul 24 20:53:04.975041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.974693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2" event={"ID":"fadfcfb2-238e-4e0e-9b00-9609f0356212","Type":"ContainerStarted","Data":"7386b2a292dcbe35b9d89c9fdbe9500de107e3493f2ff4a8236c583be7db7654"}
Jul 24 20:53:04.975041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.974750    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:04.990826 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:04.990779    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2" podStartSLOduration=0.652724167 podStartE2EDuration="990.764328ms" totalImagesPullingTime="338.040161ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:53:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:53:04.989388882 +0000 UTC m=+2259.915527785" watchObservedRunningTime="2026-07-24 20:53:04.990764328 +0000 UTC m=+2259.916903231"
Jul 24 20:53:10.963674 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:10.963634    2566 patch_prober.go:28] interesting pod/e2e-api-a-678134-openshift-default-b757565f8-v5j82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Jul 24 20:53:10.964091 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:10.963693    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Jul 24 20:53:10.986284 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:10.986259    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:12.010172 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:12.010138    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9423d64-6531-4919-b315-1af882795022" containerID="7be09376a1d25a2bf85e78ed6b58e69e7524007aff07f82e6d9dd69e656b18ea" exitCode=0
Jul 24 20:53:12.010548 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:12.010215    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" event={"ID":"d9423d64-6531-4919-b315-1af882795022","Type":"ContainerDied","Data":"7be09376a1d25a2bf85e78ed6b58e69e7524007aff07f82e6d9dd69e656b18ea"}
Jul 24 20:53:12.010619 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:12.010606    2566 scope.go:117] "RemoveContainer" containerID="7be09376a1d25a2bf85e78ed6b58e69e7524007aff07f82e6d9dd69e656b18ea"
Jul 24 20:53:12.011612 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:12.011588    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:53:12.011677 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:12.011650    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:53:12.011713 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:12.011683    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:53:13.017041 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:13.017009    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" event={"ID":"d9423d64-6531-4919-b315-1af882795022","Type":"ContainerStarted","Data":"a983b9a4121d4ea8ee196e9c15d12eb148f6dbb2a8ef8fc1695cc4dde5fc3385"}
Jul 24 20:53:13.047449 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:13.047418    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:53:13.047779 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:13.047758    2566 patch_prober.go:28] interesting pod/e2e-api-a-678134-openshift-default-b757565f8-v5j82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 20:53:13.047964 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:13.047941    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 20:53:14.047544 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:14.047491    2566 patch_prober.go:28] interesting pod/e2e-api-a-678134-openshift-default-b757565f8-v5j82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 20:53:14.048062 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:14.047570    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 20:53:15.047573 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:15.047518    2566 patch_prober.go:28] interesting pod/e2e-api-a-678134-openshift-default-b757565f8-v5j82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 20:53:15.048058 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:15.047604    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 20:53:16.046954 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.046914    2566 patch_prober.go:28] interesting pod/e2e-api-a-678134-openshift-default-b757565f8-v5j82 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused" start-of-body=
Jul 24 20:53:16.047147 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.046972    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.73:15021/healthz/ready\": dial tcp 10.133.0.73:15021: connect: connection refused"
Jul 24 20:53:16.674281 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.674248    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"]
Jul 24 20:53:16.677432 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.677410    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"
Jul 24 20:53:16.679778 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.679758    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 20:53:16.683563 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.683542    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"]
Jul 24 20:53:16.749019 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.748992    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f351a-51ee-4874-ab5f-7c93409f81dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-678134-ngjd2\" (UID: \"0e5f351a-51ee-4874-ab5f-7c93409f81dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"
Jul 24 20:53:16.850018 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.849989    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f351a-51ee-4874-ab5f-7c93409f81dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-678134-ngjd2\" (UID: \"0e5f351a-51ee-4874-ab5f-7c93409f81dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"
Jul 24 20:53:16.850650 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.850631    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f351a-51ee-4874-ab5f-7c93409f81dc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-678134-ngjd2\" (UID: \"0e5f351a-51ee-4874-ab5f-7c93409f81dc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"
Jul 24 20:53:16.989465 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:16.989407    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"
Jul 24 20:53:17.051937 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:17.051651    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:53:17.052231 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:17.052202    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:53:17.053052 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:17.053005    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:53:17.147749 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:17.147514    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"]
Jul 24 20:53:17.148252 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:53:17.148214    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e5f351a_51ee_4874_ab5f_7c93409f81dc.slice/crio-2037dcb8f01f561911b503f400dc42405d071b9e6605f1ebeb5abf8914eb9135 WatchSource:0}: Error finding container 2037dcb8f01f561911b503f400dc42405d071b9e6605f1ebeb5abf8914eb9135: Status 404 returned error can't find the container with id 2037dcb8f01f561911b503f400dc42405d071b9e6605f1ebeb5abf8914eb9135
Jul 24 20:53:18.042267 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:18.042231    2566 generic.go:356] "Generic (PLEG): container finished" podID="0e5f351a-51ee-4874-ab5f-7c93409f81dc" containerID="b151bfd69b3d24e8a0c54a6186c5ad5938d01f3d44497c4b11c7fcb8f16604c0" exitCode=0
Jul 24 20:53:18.042710 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:18.042273    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2" event={"ID":"0e5f351a-51ee-4874-ab5f-7c93409f81dc","Type":"ContainerDied","Data":"b151bfd69b3d24e8a0c54a6186c5ad5938d01f3d44497c4b11c7fcb8f16604c0"}
Jul 24 20:53:18.042710 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:18.042328    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2" event={"ID":"0e5f351a-51ee-4874-ab5f-7c93409f81dc","Type":"ContainerStarted","Data":"2037dcb8f01f561911b503f400dc42405d071b9e6605f1ebeb5abf8914eb9135"}
Jul 24 20:53:19.183325 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:19.183299    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"
Jul 24 20:53:19.271919 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:19.271888    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f351a-51ee-4874-ab5f-7c93409f81dc-maas-api-service-ca\") pod \"0e5f351a-51ee-4874-ab5f-7c93409f81dc\" (UID: \"0e5f351a-51ee-4874-ab5f-7c93409f81dc\") "
Jul 24 20:53:19.272184 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:19.272165    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e5f351a-51ee-4874-ab5f-7c93409f81dc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0e5f351a-51ee-4874-ab5f-7c93409f81dc" (UID: "0e5f351a-51ee-4874-ab5f-7c93409f81dc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:53:19.373218 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:19.373173    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0e5f351a-51ee-4874-ab5f-7c93409f81dc-maas-api-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:20.053207 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:20.053132    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"
Jul 24 20:53:20.053207 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:20.053177    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2" event={"ID":"0e5f351a-51ee-4874-ab5f-7c93409f81dc","Type":"ContainerDied","Data":"2037dcb8f01f561911b503f400dc42405d071b9e6605f1ebeb5abf8914eb9135"}
Jul 24 20:53:20.053384 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:20.053228    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2037dcb8f01f561911b503f400dc42405d071b9e6605f1ebeb5abf8914eb9135"
Jul 24 20:53:32.402383 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.402346    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"]
Jul 24 20:53:32.402860 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.402608    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb" podUID="266a5e2b-f5ad-496a-aaab-61b5bcb9481a" containerName="maas-api" containerID="cri-o://b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f" gracePeriod=30
Jul 24 20:53:32.665206 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.665146    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:53:32.786430 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.786398    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-openshift-service-ca\") pod \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") "
Jul 24 20:53:32.786615 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.786476    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c8mjf\" (UniqueName: \"kubernetes.io/projected/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-kube-api-access-c8mjf\") pod \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") "
Jul 24 20:53:32.786615 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.786492    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-maas-api-tls\") pod \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\" (UID: \"266a5e2b-f5ad-496a-aaab-61b5bcb9481a\") "
Jul 24 20:53:32.786787 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.786765    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "266a5e2b-f5ad-496a-aaab-61b5bcb9481a" (UID: "266a5e2b-f5ad-496a-aaab-61b5bcb9481a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:53:32.788476 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.788449    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-kube-api-access-c8mjf" (OuterVolumeSpecName: "kube-api-access-c8mjf") pod "266a5e2b-f5ad-496a-aaab-61b5bcb9481a" (UID: "266a5e2b-f5ad-496a-aaab-61b5bcb9481a"). InnerVolumeSpecName "kube-api-access-c8mjf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:53:32.788476 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.788455    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "266a5e2b-f5ad-496a-aaab-61b5bcb9481a" (UID: "266a5e2b-f5ad-496a-aaab-61b5bcb9481a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:53:32.887336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.887305    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:32.887336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.887329    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8mjf\" (UniqueName: \"kubernetes.io/projected/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-kube-api-access-c8mjf\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:32.887336 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:32.887339    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/266a5e2b-f5ad-496a-aaab-61b5bcb9481a-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:33.116823 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.116792    2566 generic.go:356] "Generic (PLEG): container finished" podID="266a5e2b-f5ad-496a-aaab-61b5bcb9481a" containerID="b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f" exitCode=0
Jul 24 20:53:33.116962 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.116841    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"
Jul 24 20:53:33.116962 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.116861    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb" event={"ID":"266a5e2b-f5ad-496a-aaab-61b5bcb9481a","Type":"ContainerDied","Data":"b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f"}
Jul 24 20:53:33.116962 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.116938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb" event={"ID":"266a5e2b-f5ad-496a-aaab-61b5bcb9481a","Type":"ContainerDied","Data":"fa57cf16774b409115ef83b5ec57a659733d53c812a00fb46151e601809e4bac"}
Jul 24 20:53:33.116962 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.116960    2566 scope.go:117] "RemoveContainer" containerID="b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f"
Jul 24 20:53:33.125262 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.125244    2566 scope.go:117] "RemoveContainer" containerID="b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f"
Jul 24 20:53:33.125520 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:53:33.125497    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f\": container with ID starting with b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f not found: ID does not exist" containerID="b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f"
Jul 24 20:53:33.125637 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.125541    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f"} err="failed to get container status \"b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f\": rpc error: code = NotFound desc = could not find container \"b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f\": container with ID starting with b3771057892f9b3dad0804f9f07b8985072f814c9de8a1d166ee9dbbe8991a8f not found: ID does not exist"
Jul 24 20:53:33.137151 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.137131    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"]
Jul 24 20:53:33.140888 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.140870    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-678134-7547b9d496-8fctb"]
Jul 24 20:53:33.692974 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:33.692944    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="266a5e2b-f5ad-496a-aaab-61b5bcb9481a" path="/var/lib/kubelet/pods/266a5e2b-f5ad-496a-aaab-61b5bcb9481a/volumes"
Jul 24 20:53:36.737329 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:36.737293    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"]
Jul 24 20:53:36.740044 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:36.740020    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-678134-ngjd2"]
Jul 24 20:53:37.691861 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:37.691829    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0e5f351a-51ee-4874-ab5f-7c93409f81dc" path="/var/lib/kubelet/pods/0e5f351a-51ee-4874-ab5f-7c93409f81dc/volumes"
Jul 24 20:53:43.070298 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:43.070255    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"]
Jul 24 20:53:43.070824 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:43.070510    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy" containerID="cri-o://a983b9a4121d4ea8ee196e9c15d12eb148f6dbb2a8ef8fc1695cc4dde5fc3385" gracePeriod=30
Jul 24 20:53:47.054768 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:47.054727    2566 patch_prober.go:28] interesting pod/e2e-api-a-678134-openshift-default-b757565f8-v5j82 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Jul 24 20:53:47.055127 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:47.054803    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Jul 24 20:53:48.191684 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.191658    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9423d64-6531-4919-b315-1af882795022" containerID="a983b9a4121d4ea8ee196e9c15d12eb148f6dbb2a8ef8fc1695cc4dde5fc3385" exitCode=0
Jul 24 20:53:48.192018 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.191693    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" event={"ID":"d9423d64-6531-4919-b315-1af882795022","Type":"ContainerDied","Data":"a983b9a4121d4ea8ee196e9c15d12eb148f6dbb2a8ef8fc1695cc4dde5fc3385"}
Jul 24 20:53:48.192018 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.191742    2566 scope.go:117] "RemoveContainer" containerID="7be09376a1d25a2bf85e78ed6b58e69e7524007aff07f82e6d9dd69e656b18ea"
Jul 24 20:53:48.333277 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.333224    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:53:48.519140 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519110    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-data\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519140 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519144    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-istio-token\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519161    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-envoy\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519186    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-socket\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519229    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9423d64-6531-4919-b315-1af882795022-istiod-ca-cert\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519252    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t48zp\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-kube-api-access-t48zp\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519268    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9423d64-6531-4919-b315-1af882795022-istio-podinfo\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519285    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-certs\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519355 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519332    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-credential-socket\") pod \"d9423d64-6531-4919-b315-1af882795022\" (UID: \"d9423d64-6531-4919-b315-1af882795022\") "
Jul 24 20:53:48.519794 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519768    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:53:48.519981 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9423d64-6531-4919-b315-1af882795022-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:53:48.519981 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519966    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:53:48.520148 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.519876    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:53:48.520148 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.520028    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-data" (OuterVolumeSpecName: "istio-data") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:53:48.521408 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.521378    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:53:48.521485 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.521423    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-istio-token" (OuterVolumeSpecName: "istio-token") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:53:48.521485 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.521432    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-kube-api-access-t48zp" (OuterVolumeSpecName: "kube-api-access-t48zp") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "kube-api-access-t48zp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:53:48.521617 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.521574    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9423d64-6531-4919-b315-1af882795022-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d9423d64-6531-4919-b315-1af882795022" (UID: "d9423d64-6531-4919-b315-1af882795022"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 20:53:48.620074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620050    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-envoy\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620074 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620074    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620083    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9423d64-6531-4919-b315-1af882795022-istiod-ca-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620092    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t48zp\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-kube-api-access-t48zp\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620101    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9423d64-6531-4919-b315-1af882795022-istio-podinfo\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620110    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-workload-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620119    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-credential-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620126    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9423d64-6531-4919-b315-1af882795022-istio-data\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:48.620243 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:48.620134    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9423d64-6531-4919-b315-1af882795022-istio-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:49.198150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:49.198122    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"
Jul 24 20:53:49.198150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:49.198138    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82" event={"ID":"d9423d64-6531-4919-b315-1af882795022","Type":"ContainerDied","Data":"9827257b3f90aa6684bdbd3424a95e5e83c38339bea9c21f047e7215149cbc52"}
Jul 24 20:53:49.198671 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:49.198195    2566 scope.go:117] "RemoveContainer" containerID="a983b9a4121d4ea8ee196e9c15d12eb148f6dbb2a8ef8fc1695cc4dde5fc3385"
Jul 24 20:53:49.218934 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:49.218912    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"]
Jul 24 20:53:49.223140 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:49.223120    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-678134-openshift-default-b757565f8-v5j82"]
Jul 24 20:53:49.692543 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:49.692504    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9423d64-6531-4919-b315-1af882795022" path="/var/lib/kubelet/pods/d9423d64-6531-4919-b315-1af882795022/volumes"
Jul 24 20:53:58.843482 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:58.843445    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"]
Jul 24 20:53:58.843906 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:58.843749    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2" podUID="fadfcfb2-238e-4e0e-9b00-9609f0356212" containerName="maas-api" containerID="cri-o://198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae" gracePeriod=30
Jul 24 20:53:59.097908 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.097885    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:59.103588 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.103567    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfcfb2-238e-4e0e-9b00-9609f0356212-openshift-service-ca\") pod \"fadfcfb2-238e-4e0e-9b00-9609f0356212\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") "
Jul 24 20:53:59.103698 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.103598    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxzb5\" (UniqueName: \"kubernetes.io/projected/fadfcfb2-238e-4e0e-9b00-9609f0356212-kube-api-access-xxzb5\") pod \"fadfcfb2-238e-4e0e-9b00-9609f0356212\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") "
Jul 24 20:53:59.103956 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.103933    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fadfcfb2-238e-4e0e-9b00-9609f0356212-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fadfcfb2-238e-4e0e-9b00-9609f0356212" (UID: "fadfcfb2-238e-4e0e-9b00-9609f0356212"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 20:53:59.105485 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.105467    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fadfcfb2-238e-4e0e-9b00-9609f0356212-kube-api-access-xxzb5" (OuterVolumeSpecName: "kube-api-access-xxzb5") pod "fadfcfb2-238e-4e0e-9b00-9609f0356212" (UID: "fadfcfb2-238e-4e0e-9b00-9609f0356212"). InnerVolumeSpecName "kube-api-access-xxzb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:53:59.204773 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.204747    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfcfb2-238e-4e0e-9b00-9609f0356212-maas-api-tls\") pod \"fadfcfb2-238e-4e0e-9b00-9609f0356212\" (UID: \"fadfcfb2-238e-4e0e-9b00-9609f0356212\") "
Jul 24 20:53:59.205021 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.205004    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fadfcfb2-238e-4e0e-9b00-9609f0356212-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:59.205095 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.205026    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxzb5\" (UniqueName: \"kubernetes.io/projected/fadfcfb2-238e-4e0e-9b00-9609f0356212-kube-api-access-xxzb5\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:59.206822 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.206799    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fadfcfb2-238e-4e0e-9b00-9609f0356212-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fadfcfb2-238e-4e0e-9b00-9609f0356212" (UID: "fadfcfb2-238e-4e0e-9b00-9609f0356212"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:53:59.249065 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.249037    2566 generic.go:356] "Generic (PLEG): container finished" podID="fadfcfb2-238e-4e0e-9b00-9609f0356212" containerID="198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae" exitCode=0
Jul 24 20:53:59.249183 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.249096    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"
Jul 24 20:53:59.249183 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.249118    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2" event={"ID":"fadfcfb2-238e-4e0e-9b00-9609f0356212","Type":"ContainerDied","Data":"198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae"}
Jul 24 20:53:59.249183 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.249172    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2" event={"ID":"fadfcfb2-238e-4e0e-9b00-9609f0356212","Type":"ContainerDied","Data":"7386b2a292dcbe35b9d89c9fdbe9500de107e3493f2ff4a8236c583be7db7654"}
Jul 24 20:53:59.249330 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.249193    2566 scope.go:117] "RemoveContainer" containerID="198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae"
Jul 24 20:53:59.257600 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.257584    2566 scope.go:117] "RemoveContainer" containerID="198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae"
Jul 24 20:53:59.257841 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:53:59.257823    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae\": container with ID starting with 198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae not found: ID does not exist" containerID="198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae"
Jul 24 20:53:59.257901 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.257848    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae"} err="failed to get container status \"198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae\": rpc error: code = NotFound desc = could not find container \"198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae\": container with ID starting with 198b74f091ee612cc5bdefc86289dcdab16a2444a9da829d8399c02170b2d1ae not found: ID does not exist"
Jul 24 20:53:59.269169 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.269145    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"]
Jul 24 20:53:59.277376 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.277353    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-1ae1a9-547495d6db-r27x2"]
Jul 24 20:53:59.305972 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.305952    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fadfcfb2-238e-4e0e-9b00-9609f0356212-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:53:59.691711 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:53:59.691681    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fadfcfb2-238e-4e0e-9b00-9609f0356212" path="/var/lib/kubelet/pods/fadfcfb2-238e-4e0e-9b00-9609f0356212/volumes"
Jul 24 20:54:10.627856 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.627822    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"]
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628151    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fadfcfb2-238e-4e0e-9b00-9609f0356212" containerName="maas-api"
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628161    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fadfcfb2-238e-4e0e-9b00-9609f0356212" containerName="maas-api"
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628174    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="266a5e2b-f5ad-496a-aaab-61b5bcb9481a" containerName="maas-api"
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628179    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="266a5e2b-f5ad-496a-aaab-61b5bcb9481a" containerName="maas-api"
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628185    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy"
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628191    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy"
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628198    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0e5f351a-51ee-4874-ab5f-7c93409f81dc" containerName="revoke-keys"
Jul 24 20:54:10.628214 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628203    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0e5f351a-51ee-4874-ab5f-7c93409f81dc" containerName="revoke-keys"
Jul 24 20:54:10.628583 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628291    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="266a5e2b-f5ad-496a-aaab-61b5bcb9481a" containerName="maas-api"
Jul 24 20:54:10.628583 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628306    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy"
Jul 24 20:54:10.628583 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628316    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0e5f351a-51ee-4874-ab5f-7c93409f81dc" containerName="revoke-keys"
Jul 24 20:54:10.628583 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628325    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fadfcfb2-238e-4e0e-9b00-9609f0356212" containerName="maas-api"
Jul 24 20:54:10.628583 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.628329    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy"
Jul 24 20:54:10.631339 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.631324    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.633765 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.633749    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-3f62e6-openshift-default-dockercfg-6qbqs\""
Jul 24 20:54:10.642035 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.642013    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"]
Jul 24 20:54:10.694977 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.694950    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-certs\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.694987    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-socket\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.695007    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70179b0f-2291-428e-ac07-04c10c47cf88-istio-podinfo\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.695023    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-credential-socket\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695086 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.695074    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-data\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.695133    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70179b0f-2291-428e-ac07-04c10c47cf88-istiod-ca-cert\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.695166    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sn76s\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-kube-api-access-sn76s\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695225 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.695211    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-envoy\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.695320 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.695230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-istio-token\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.795914 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.795883    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70179b0f-2291-428e-ac07-04c10c47cf88-istiod-ca-cert\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796064 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.795928    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sn76s\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-kube-api-access-sn76s\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796064 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796049    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-envoy\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796158 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796090    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-istio-token\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796158 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796142    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-certs\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796266 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796193    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-socket\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796322 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796282    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70179b0f-2291-428e-ac07-04c10c47cf88-istio-podinfo\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796377 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796320    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-credential-socket\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796377 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796345    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-data\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796607 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796580    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-certs\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796686 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796660    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-credential-socket\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796739 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796692    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70179b0f-2291-428e-ac07-04c10c47cf88-istiod-ca-cert\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796739 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796712    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-data\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.796848 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.796780    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-socket\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.798410 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.798387    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-envoy\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.798602 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.798584    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70179b0f-2291-428e-ac07-04c10c47cf88-istio-podinfo\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.804636 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.804615    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-istio-token\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.804713 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.804651    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sn76s\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-kube-api-access-sn76s\") pod \"e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") " pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:10.942992 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:10.942929    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:11.075788 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.075640    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"]
Jul 24 20:54:11.078016 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:54:11.077986    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70179b0f_2291_428e_ac07_04c10c47cf88.slice/crio-bcea5f99db22b4005f3ad1ab582d2020ba57865e0e7b36f8550fb64c3dd372e0 WatchSource:0}: Error finding container bcea5f99db22b4005f3ad1ab582d2020ba57865e0e7b36f8550fb64c3dd372e0: Status 404 returned error can't find the container with id bcea5f99db22b4005f3ad1ab582d2020ba57865e0e7b36f8550fb64c3dd372e0
Jul 24 20:54:11.080240 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.080210    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:54:11.080323 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.080269    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:54:11.080323 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.080295    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:54:11.307879 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.307793    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" event={"ID":"70179b0f-2291-428e-ac07-04c10c47cf88","Type":"ContainerStarted","Data":"303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43"}
Jul 24 20:54:11.307879 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.307849    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" event={"ID":"70179b0f-2291-428e-ac07-04c10c47cf88","Type":"ContainerStarted","Data":"bcea5f99db22b4005f3ad1ab582d2020ba57865e0e7b36f8550fb64c3dd372e0"}
Jul 24 20:54:11.332489 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.332432    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" podStartSLOduration=1.3324134 podStartE2EDuration="1.3324134s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:54:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:54:11.331724463 +0000 UTC m=+2326.257863388" watchObservedRunningTime="2026-07-24 20:54:11.3324134 +0000 UTC m=+2326.258552305"
Jul 24 20:54:11.943825 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:11.943793    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:12.948739 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:12.948714    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:13.317361 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.317287    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:13.318165 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.318148    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:13.357254 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.357221    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"]
Jul 24 20:54:13.357668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.357651    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy"
Jul 24 20:54:13.357668 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.357670    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9423d64-6531-4919-b315-1af882795022" containerName="istio-proxy"
Jul 24 20:54:13.361000 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.360980    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.363502 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.363471    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-3f62e6-serving-cert\""
Jul 24 20:54:13.373026 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.373007    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"]
Jul 24 20:54:13.416659 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.416629    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.416659 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.416656    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de257412-d536-45ac-9fbe-2aa5135ff724-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.416911 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.416846    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5c2c\" (UniqueName: \"kubernetes.io/projected/de257412-d536-45ac-9fbe-2aa5135ff724-kube-api-access-x5c2c\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.517730 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.517693    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5c2c\" (UniqueName: \"kubernetes.io/projected/de257412-d536-45ac-9fbe-2aa5135ff724-kube-api-access-x5c2c\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.517898 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.517773    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.517898 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.517792    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de257412-d536-45ac-9fbe-2aa5135ff724-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.517970 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:54:13.517906    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-serving-cert: secret "maas-api-e2e-shared-a-3f62e6-serving-cert" not found
Jul 24 20:54:13.518008 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:54:13.517981    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls podName:de257412-d536-45ac-9fbe-2aa5135ff724 nodeName:}" failed. No retries permitted until 2026-07-24 20:54:14.017959085 +0000 UTC m=+2328.944097986 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls") pod "maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd" (UID: "de257412-d536-45ac-9fbe-2aa5135ff724") : secret "maas-api-e2e-shared-a-3f62e6-serving-cert" not found
Jul 24 20:54:13.518444 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.518426    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de257412-d536-45ac-9fbe-2aa5135ff724-openshift-service-ca\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:13.527598 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:13.527574    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5c2c\" (UniqueName: \"kubernetes.io/projected/de257412-d536-45ac-9fbe-2aa5135ff724-kube-api-access-x5c2c\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:14.021806 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:14.021774    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:14.024206 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:14.024181    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls\") pod \"maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:14.276548 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:14.276435    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:14.413007 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:14.412982    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"]
Jul 24 20:54:14.414883 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:54:14.414852    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde257412_d536_45ac_9fbe_2aa5135ff724.slice/crio-06040d751c4517c7de43f4b79939500a6e25d67dd80e6da6721a402245e0755d WatchSource:0}: Error finding container 06040d751c4517c7de43f4b79939500a6e25d67dd80e6da6721a402245e0755d: Status 404 returned error can't find the container with id 06040d751c4517c7de43f4b79939500a6e25d67dd80e6da6721a402245e0755d
Jul 24 20:54:15.329163 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:15.329063    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd" event={"ID":"de257412-d536-45ac-9fbe-2aa5135ff724","Type":"ContainerStarted","Data":"235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635"}
Jul 24 20:54:15.329163 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:15.329133    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd" event={"ID":"de257412-d536-45ac-9fbe-2aa5135ff724","Type":"ContainerStarted","Data":"06040d751c4517c7de43f4b79939500a6e25d67dd80e6da6721a402245e0755d"}
Jul 24 20:54:15.329568 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:15.329242    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:15.349590 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:15.349521    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd" podStartSLOduration=1.915958685 podStartE2EDuration="2.349505988s" totalImagesPullingTime="433.547303ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:54:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:54:15.34756487 +0000 UTC m=+2330.273703803" watchObservedRunningTime="2026-07-24 20:54:15.349505988 +0000 UTC m=+2330.275644891"
Jul 24 20:54:21.340059 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.339980    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 20:54:21.550984 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.550947    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"]
Jul 24 20:54:21.557989 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.557952    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.562837 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.562781    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-449ecf-serving-cert\""
Jul 24 20:54:21.566505 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.566479    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"]
Jul 24 20:54:21.593591 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.593511    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bg6z\" (UniqueName: \"kubernetes.io/projected/d9884d50-47a7-4a93-a9d8-96a735c0c90c-kube-api-access-5bg6z\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.593708 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.593605    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.593708 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.593632    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9884d50-47a7-4a93-a9d8-96a735c0c90c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.694903 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.694874    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bg6z\" (UniqueName: \"kubernetes.io/projected/d9884d50-47a7-4a93-a9d8-96a735c0c90c-kube-api-access-5bg6z\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.695089 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.694912    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.695089 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.694933    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9884d50-47a7-4a93-a9d8-96a735c0c90c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.695089 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:54:21.695044    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-serving-cert: secret "maas-api-e2e-shared-b-449ecf-serving-cert" not found
Jul 24 20:54:21.695260 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:54:21.695110    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls podName:d9884d50-47a7-4a93-a9d8-96a735c0c90c nodeName:}" failed. No retries permitted until 2026-07-24 20:54:22.195088824 +0000 UTC m=+2337.121227707 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls") pod "maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8" (UID: "d9884d50-47a7-4a93-a9d8-96a735c0c90c") : secret "maas-api-e2e-shared-b-449ecf-serving-cert" not found
Jul 24 20:54:21.695570 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.695552    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9884d50-47a7-4a93-a9d8-96a735c0c90c-openshift-service-ca\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:21.703846 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:21.703828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bg6z\" (UniqueName: \"kubernetes.io/projected/d9884d50-47a7-4a93-a9d8-96a735c0c90c-kube-api-access-5bg6z\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:22.199603 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:22.199571    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:22.202574 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:22.202519    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls\") pod \"maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:22.482066 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:22.481975    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:22.659866 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:22.646949    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"]
Jul 24 20:54:23.377861 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:23.377826    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8" event={"ID":"d9884d50-47a7-4a93-a9d8-96a735c0c90c","Type":"ContainerStarted","Data":"32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0"}
Jul 24 20:54:23.378019 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:23.377885    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8" event={"ID":"d9884d50-47a7-4a93-a9d8-96a735c0c90c","Type":"ContainerStarted","Data":"ee5b80bd80966ac44fb9a7d30509f009c2d6ce203520bef1f862d92209a99a7d"}
Jul 24 20:54:23.378062 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:23.378040    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:23.396356 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:23.396306    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8" podStartSLOduration=2.080875212 podStartE2EDuration="2.396291831s" totalImagesPullingTime="315.416619ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:54:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:54:23.394627442 +0000 UTC m=+2338.320766366" watchObservedRunningTime="2026-07-24 20:54:23.396291831 +0000 UTC m=+2338.322430734"
Jul 24 20:54:28.318499 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.318419    2566 patch_prober.go:28] interesting pod/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 20:54:28.318499 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.318489    2566 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 20:54:28.403484 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.403446    2566 generic.go:356] "Generic (PLEG): container finished" podID="70179b0f-2291-428e-ac07-04c10c47cf88" containerID="303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43" exitCode=0
Jul 24 20:54:28.403646 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.403519    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" event={"ID":"70179b0f-2291-428e-ac07-04c10c47cf88","Type":"ContainerDied","Data":"303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43"}
Jul 24 20:54:28.403911 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.403897    2566 scope.go:117] "RemoveContainer" containerID="303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43"
Jul 24 20:54:28.404872 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.404844    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:54:28.404938 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.404924    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:54:28.404980 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:28.404966    2566 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236228Ki","pods":"250"}
Jul 24 20:54:29.388641 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:29.388610    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 20:54:29.424772 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:29.424732    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" event={"ID":"70179b0f-2291-428e-ac07-04c10c47cf88","Type":"ContainerStarted","Data":"b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4"}
Jul 24 20:54:29.944199 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:29.944156    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:29.944410 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:29.944309    2566 patch_prober.go:28] interesting pod/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 20:54:29.944410 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:29.944353    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 20:54:30.944310 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:30.944273    2566 patch_prober.go:28] interesting pod/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 20:54:30.944826 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:30.944361    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 20:54:31.944358 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:31.943942    2566 patch_prober.go:28] interesting pod/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 20:54:31.944358 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:31.943999    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 20:54:32.944379 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:32.944341    2566 patch_prober.go:28] interesting pod/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body=
Jul 24 20:54:32.944852 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:32.944409    2566 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused"
Jul 24 20:54:33.947870 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:33.947841    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:33.948246 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:33.948100    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:33.948775 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:33.948757    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 20:54:35.391882 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.391845    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"]
Jul 24 20:54:35.395360 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.395339    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.399165 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.399143    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f62e6\"/\"openshift-service-ca.crt\""
Jul 24 20:54:35.399284 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.399143    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f62e6\"/\"default-dockercfg-v2ttx\""
Jul 24 20:54:35.399284 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.399222    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-3f62e6\"/\"kube-root-ca.crt\""
Jul 24 20:54:35.399383 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.399301    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f62e6\"/\"auth-test-model-3f62e6-kserve-self-signed-certs\""
Jul 24 20:54:35.404748 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.404728    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"]
Jul 24 20:54:35.521227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.521200    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tmp-dir\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.521227 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.521230    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-dshm\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.521448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.521262    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-home\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.521448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.521282    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx5pb\" (UniqueName: \"kubernetes.io/projected/30460235-ec3e-4a2d-be9c-8c8a14b9891a-kube-api-access-hx5pb\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.521448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.521377    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tls-certs\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.521448 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.521415    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-model-cache\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.622587 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.622561    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tmp-dir\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.622718 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.622592    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-dshm\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.622718 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.622612    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-home\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.622832 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.622727    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx5pb\" (UniqueName: \"kubernetes.io/projected/30460235-ec3e-4a2d-be9c-8c8a14b9891a-kube-api-access-hx5pb\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.622832 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.622803    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tls-certs\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.622932 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.622856    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-model-cache\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.622990 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.622961    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tmp-dir\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.623048 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.623029    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-home\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.623197 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.623179    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-model-cache\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.624638 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.624621    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-dshm\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.625107 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.625088    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tls-certs\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.631003 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.630975    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx5pb\" (UniqueName: \"kubernetes.io/projected/30460235-ec3e-4a2d-be9c-8c8a14b9891a-kube-api-access-hx5pb\") pod \"auth-test-model-3f62e6-kserve-599689b99-dxbgg\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") " pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.704743 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.704658    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:35.834590 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:35.834567    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"]
Jul 24 20:54:35.836830 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:54:35.836798    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod30460235_ec3e_4a2d_be9c_8c8a14b9891a.slice/crio-ed403f09832ab6849a564ff7e9295c50e28808f0516b1a0f0e7bc6b8b2031d82 WatchSource:0}: Error finding container ed403f09832ab6849a564ff7e9295c50e28808f0516b1a0f0e7bc6b8b2031d82: Status 404 returned error can't find the container with id ed403f09832ab6849a564ff7e9295c50e28808f0516b1a0f0e7bc6b8b2031d82
Jul 24 20:54:36.463667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:36.463624    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg" event={"ID":"30460235-ec3e-4a2d-be9c-8c8a14b9891a","Type":"ContainerStarted","Data":"4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734"}
Jul 24 20:54:36.463667 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:36.463668    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg" event={"ID":"30460235-ec3e-4a2d-be9c-8c8a14b9891a","Type":"ContainerStarted","Data":"ed403f09832ab6849a564ff7e9295c50e28808f0516b1a0f0e7bc6b8b2031d82"}
Jul 24 20:54:36.479493 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:36.479448    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg" podStartSLOduration=1.479433358 podStartE2EDuration="1.479433358s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:54:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:54:36.479027693 +0000 UTC m=+2351.405166596" watchObservedRunningTime="2026-07-24 20:54:36.479433358 +0000 UTC m=+2351.405572262"
Jul 24 20:54:45.704899 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:45.704863    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:45.705321 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:45.704916    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:45.717230 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:45.717206    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:54:46.522715 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:54:46.522690    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:55:25.831815 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:55:25.831787    2566 scope.go:117] "RemoveContainer" containerID="4a64187edf99160baaf01506bf5582012e6b53d0f6464bc6c7c344d071e28a75"
Jul 24 20:59:00.007122 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:00.007087    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"]
Jul 24 20:59:10.996162 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:10.996128    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"]
Jul 24 20:59:10.999491 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:10.999476    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.002103 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.002085    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-3f62e6\"/\"sub-test-model-3f62e6-kserve-self-signed-certs\""
Jul 24 20:59:11.007833 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.007809    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"]
Jul 24 20:59:11.075447 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.075413    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-dshm\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.075654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.075481    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tls-certs\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.075654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.075558    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-model-cache\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.075654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.075584    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-home\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.075654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.075614    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tmp-dir\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.075654 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.075646    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pmn64\" (UniqueName: \"kubernetes.io/projected/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-kube-api-access-pmn64\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.176632 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.176601    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-dshm\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.176817 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.176648    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tls-certs\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.176817 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.176787    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-model-cache\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.176948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.176832    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-home\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.176948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.176867    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tmp-dir\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.176948 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.176901    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pmn64\" (UniqueName: \"kubernetes.io/projected/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-kube-api-access-pmn64\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.177216 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.177187    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-model-cache\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.177349 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.177228    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-home\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.177349 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.177276    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tmp-dir\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.178944 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.178916    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-dshm\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.179088 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.179071    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tls-certs\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.186480 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.186459    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pmn64\" (UniqueName: \"kubernetes.io/projected/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-kube-api-access-pmn64\") pod \"sub-test-model-3f62e6-kserve-687fdb95bc-wlnws\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") " pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.310077 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.309997    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:11.445490 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.445466    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"]
Jul 24 20:59:11.447189 ip-10-0-129-245 kubenswrapper[2566]: W0724 20:59:11.447160    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9d02ac4_c855_4c36_ad77_7ad27b4e571b.slice/crio-740ee11ef4ac4aa1bd6b46169cc5dbbabe558cfba1723df115098fcf029d3631 WatchSource:0}: Error finding container 740ee11ef4ac4aa1bd6b46169cc5dbbabe558cfba1723df115098fcf029d3631: Status 404 returned error can't find the container with id 740ee11ef4ac4aa1bd6b46169cc5dbbabe558cfba1723df115098fcf029d3631
Jul 24 20:59:11.787265 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.787236    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws" event={"ID":"f9d02ac4-c855-4c36-ad77-7ad27b4e571b","Type":"ContainerStarted","Data":"9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7"}
Jul 24 20:59:11.787420 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.787277    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws" event={"ID":"f9d02ac4-c855-4c36-ad77-7ad27b4e571b","Type":"ContainerStarted","Data":"740ee11ef4ac4aa1bd6b46169cc5dbbabe558cfba1723df115098fcf029d3631"}
Jul 24 20:59:11.804521 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:11.804475    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws" podStartSLOduration=1.804460228 podStartE2EDuration="1.804460228s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 20:59:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 20:59:11.8026457 +0000 UTC m=+2626.728784616" watchObservedRunningTime="2026-07-24 20:59:11.804460228 +0000 UTC m=+2626.730599182"
Jul 24 20:59:15.025424 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.025385    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg" podUID="30460235-ec3e-4a2d-be9c-8c8a14b9891a" containerName="main" containerID="cri-o://4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734" gracePeriod=45
Jul 24 20:59:15.281372 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.281311    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:59:15.313650 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.313622    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tmp-dir\") pod \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") "
Jul 24 20:59:15.313820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.313684    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-dshm\") pod \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") "
Jul 24 20:59:15.313820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.313729    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-home\") pod \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") "
Jul 24 20:59:15.313820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.313760    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hx5pb\" (UniqueName: \"kubernetes.io/projected/30460235-ec3e-4a2d-be9c-8c8a14b9891a-kube-api-access-hx5pb\") pod \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") "
Jul 24 20:59:15.313820 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.313780    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tls-certs\") pod \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") "
Jul 24 20:59:15.314037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.313845    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-model-cache\") pod \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\" (UID: \"30460235-ec3e-4a2d-be9c-8c8a14b9891a\") "
Jul 24 20:59:15.314037 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.313925    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "30460235-ec3e-4a2d-be9c-8c8a14b9891a" (UID: "30460235-ec3e-4a2d-be9c-8c8a14b9891a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:59:15.314144 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.314045    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-home" (OuterVolumeSpecName: "home") pod "30460235-ec3e-4a2d-be9c-8c8a14b9891a" (UID: "30460235-ec3e-4a2d-be9c-8c8a14b9891a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:59:15.314187 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.314137    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-model-cache" (OuterVolumeSpecName: "model-cache") pod "30460235-ec3e-4a2d-be9c-8c8a14b9891a" (UID: "30460235-ec3e-4a2d-be9c-8c8a14b9891a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:59:15.314248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.314226    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-model-cache\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:59:15.314248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.314238    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tmp-dir\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:59:15.314248 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.314247    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-home\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:59:15.316005 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.315983    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-dshm" (OuterVolumeSpecName: "dshm") pod "30460235-ec3e-4a2d-be9c-8c8a14b9891a" (UID: "30460235-ec3e-4a2d-be9c-8c8a14b9891a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 20:59:15.316126 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.316001    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "30460235-ec3e-4a2d-be9c-8c8a14b9891a" (UID: "30460235-ec3e-4a2d-be9c-8c8a14b9891a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 20:59:15.316126 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.316107    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30460235-ec3e-4a2d-be9c-8c8a14b9891a-kube-api-access-hx5pb" (OuterVolumeSpecName: "kube-api-access-hx5pb") pod "30460235-ec3e-4a2d-be9c-8c8a14b9891a" (UID: "30460235-ec3e-4a2d-be9c-8c8a14b9891a"). InnerVolumeSpecName "kube-api-access-hx5pb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 20:59:15.415263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.415235    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/30460235-ec3e-4a2d-be9c-8c8a14b9891a-dshm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:59:15.415263 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.415261    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hx5pb\" (UniqueName: \"kubernetes.io/projected/30460235-ec3e-4a2d-be9c-8c8a14b9891a-kube-api-access-hx5pb\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:59:15.415451 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.415272    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/30460235-ec3e-4a2d-be9c-8c8a14b9891a-tls-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 20:59:15.808149 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.808118    2566 generic.go:356] "Generic (PLEG): container finished" podID="30460235-ec3e-4a2d-be9c-8c8a14b9891a" containerID="4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734" exitCode=0
Jul 24 20:59:15.808318 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.808162    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"
Jul 24 20:59:15.808318 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.808196    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg" event={"ID":"30460235-ec3e-4a2d-be9c-8c8a14b9891a","Type":"ContainerDied","Data":"4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734"}
Jul 24 20:59:15.808318 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.808228    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg" event={"ID":"30460235-ec3e-4a2d-be9c-8c8a14b9891a","Type":"ContainerDied","Data":"ed403f09832ab6849a564ff7e9295c50e28808f0516b1a0f0e7bc6b8b2031d82"}
Jul 24 20:59:15.808318 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.808243    2566 scope.go:117] "RemoveContainer" containerID="4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734"
Jul 24 20:59:15.816606 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.816582    2566 scope.go:117] "RemoveContainer" containerID="4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734"
Jul 24 20:59:15.816838 ip-10-0-129-245 kubenswrapper[2566]: E0724 20:59:15.816814    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734\": container with ID starting with 4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734 not found: ID does not exist" containerID="4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734"
Jul 24 20:59:15.816944 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.816843    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734"} err="failed to get container status \"4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734\": rpc error: code = NotFound desc = could not find container \"4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734\": container with ID starting with 4507c9d47188a9f9e509e98acabbac34bb38de318a9f2cc5d115e5626a289734 not found: ID does not exist"
Jul 24 20:59:15.825321 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.825299    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"]
Jul 24 20:59:15.828818 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:15.828799    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/auth-test-model-3f62e6-kserve-599689b99-dxbgg"]
Jul 24 20:59:17.693613 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:17.693580    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30460235-ec3e-4a2d-be9c-8c8a14b9891a" path="/var/lib/kubelet/pods/30460235-ec3e-4a2d-be9c-8c8a14b9891a/volumes"
Jul 24 20:59:21.310150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:21.310109    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:21.310150 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:21.310164    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:21.322601 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:21.322579    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:21.852708 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:21.852681    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 20:59:25.871967 ip-10-0-129-245 kubenswrapper[2566]: I0724 20:59:25.871937    2566 scope.go:117] "RemoveContainer" containerID="b151bfd69b3d24e8a0c54a6186c5ad5938d01f3d44497c4b11c7fcb8f16604c0"
Jul 24 21:00:00.131616 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.131579    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c"]
Jul 24 21:00:00.133899 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.131948    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30460235-ec3e-4a2d-be9c-8c8a14b9891a" containerName="main"
Jul 24 21:00:00.133899 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.131958    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="30460235-ec3e-4a2d-be9c-8c8a14b9891a" containerName="main"
Jul 24 21:00:00.133899 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.132032    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="30460235-ec3e-4a2d-be9c-8c8a14b9891a" containerName="main"
Jul 24 21:00:00.134815 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.134800    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c"
Jul 24 21:00:00.137256 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.137229    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 21:00:00.142180 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.142157    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c"]
Jul 24 21:00:00.148035 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.148010    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c"
Jul 24 21:00:00.283956 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:00.283922    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c"]
Jul 24 21:00:00.285715 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:00:00.285684    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24bbe9c0_8eb4_41f9_a872_2a9b64861fed.slice/crio-b249e5ac045832e6908ef1247a160325b271405a8452314653d7f8dc0b780c74 WatchSource:0}: Error finding container b249e5ac045832e6908ef1247a160325b271405a8452314653d7f8dc0b780c74: Status 404 returned error can't find the container with id b249e5ac045832e6908ef1247a160325b271405a8452314653d7f8dc0b780c74
Jul 24 21:00:01.032146 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:01.032107    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c" event={"ID":"24bbe9c0-8eb4-41f9-a872-2a9b64861fed","Type":"ContainerStarted","Data":"6306d75c8293cffd41256fd0cd177d52ebcf8b92097134f55093b8dd8c827436"}
Jul 24 21:00:01.032360 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:01.032159    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c" event={"ID":"24bbe9c0-8eb4-41f9-a872-2a9b64861fed","Type":"ContainerStarted","Data":"b249e5ac045832e6908ef1247a160325b271405a8452314653d7f8dc0b780c74"}
Jul 24 21:00:01.050219 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:01.050164    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c" podStartSLOduration=1.050148977 podStartE2EDuration="1.050148977s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:00:01.047937518 +0000 UTC m=+2675.974076435" watchObservedRunningTime="2026-07-24 21:00:01.050148977 +0000 UTC m=+2675.976287919"
Jul 24 21:00:02.039124 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:02.039089    2566 generic.go:356] "Generic (PLEG): container finished" podID="24bbe9c0-8eb4-41f9-a872-2a9b64861fed" containerID="6306d75c8293cffd41256fd0cd177d52ebcf8b92097134f55093b8dd8c827436" exitCode=0
Jul 24 21:00:02.039550 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:02.039180    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c" event={"ID":"24bbe9c0-8eb4-41f9-a872-2a9b64861fed","Type":"ContainerDied","Data":"6306d75c8293cffd41256fd0cd177d52ebcf8b92097134f55093b8dd8c827436"}
Jul 24 21:00:03.177749 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:03.177725    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c"
Jul 24 21:00:04.052969 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:04.052893    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c"
Jul 24 21:00:04.052969 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:04.052905    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29748780-9dg5c" event={"ID":"24bbe9c0-8eb4-41f9-a872-2a9b64861fed","Type":"ContainerDied","Data":"b249e5ac045832e6908ef1247a160325b271405a8452314653d7f8dc0b780c74"}
Jul 24 21:00:04.052969 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:04.052958    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b249e5ac045832e6908ef1247a160325b271405a8452314653d7f8dc0b780c74"
Jul 24 21:00:32.501355 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:32.501267    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"]
Jul 24 21:00:47.520021 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.519949    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws" podUID="f9d02ac4-c855-4c36-ad77-7ad27b4e571b" containerName="main" containerID="cri-o://9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7" gracePeriod=45
Jul 24 21:00:47.781081 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.781003    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 21:00:47.916627 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916599    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tls-certs\") pod \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") "
Jul 24 21:00:47.916808 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916640    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-dshm\") pod \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") "
Jul 24 21:00:47.916808 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916671    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-model-cache\") pod \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") "
Jul 24 21:00:47.916808 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916690    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tmp-dir\") pod \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") "
Jul 24 21:00:47.916808 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916732    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-home\") pod \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") "
Jul 24 21:00:47.917034 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916828    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pmn64\" (UniqueName: \"kubernetes.io/projected/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-kube-api-access-pmn64\") pod \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\" (UID: \"f9d02ac4-c855-4c36-ad77-7ad27b4e571b\") "
Jul 24 21:00:47.917034 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916979    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9d02ac4-c855-4c36-ad77-7ad27b4e571b" (UID: "f9d02ac4-c855-4c36-ad77-7ad27b4e571b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:00:47.917034 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.916994    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9d02ac4-c855-4c36-ad77-7ad27b4e571b" (UID: "f9d02ac4-c855-4c36-ad77-7ad27b4e571b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:00:47.917200 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.917038    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-home" (OuterVolumeSpecName: "home") pod "f9d02ac4-c855-4c36-ad77-7ad27b4e571b" (UID: "f9d02ac4-c855-4c36-ad77-7ad27b4e571b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:00:47.917200 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.917156    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-home\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:00:47.917200 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.917178    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-model-cache\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:00:47.917200 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.917194    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tmp-dir\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:00:47.918792 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.918763    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9d02ac4-c855-4c36-ad77-7ad27b4e571b" (UID: "f9d02ac4-c855-4c36-ad77-7ad27b4e571b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:00:47.918792 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.918778    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-dshm" (OuterVolumeSpecName: "dshm") pod "f9d02ac4-c855-4c36-ad77-7ad27b4e571b" (UID: "f9d02ac4-c855-4c36-ad77-7ad27b4e571b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:00:47.918962 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:47.918875    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-kube-api-access-pmn64" (OuterVolumeSpecName: "kube-api-access-pmn64") pod "f9d02ac4-c855-4c36-ad77-7ad27b4e571b" (UID: "f9d02ac4-c855-4c36-ad77-7ad27b4e571b"). InnerVolumeSpecName "kube-api-access-pmn64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:00:48.018278 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.018241    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pmn64\" (UniqueName: \"kubernetes.io/projected/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-kube-api-access-pmn64\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:00:48.018278 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.018272    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-tls-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:00:48.018278 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.018282    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d02ac4-c855-4c36-ad77-7ad27b4e571b-dshm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:00:48.270281 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.270251    2566 generic.go:356] "Generic (PLEG): container finished" podID="f9d02ac4-c855-4c36-ad77-7ad27b4e571b" containerID="9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7" exitCode=0
Jul 24 21:00:48.270472 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.270292    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"
Jul 24 21:00:48.270472 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.270341    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws" event={"ID":"f9d02ac4-c855-4c36-ad77-7ad27b4e571b","Type":"ContainerDied","Data":"9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7"}
Jul 24 21:00:48.270472 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.270384    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws" event={"ID":"f9d02ac4-c855-4c36-ad77-7ad27b4e571b","Type":"ContainerDied","Data":"740ee11ef4ac4aa1bd6b46169cc5dbbabe558cfba1723df115098fcf029d3631"}
Jul 24 21:00:48.270472 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.270402    2566 scope.go:117] "RemoveContainer" containerID="9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7"
Jul 24 21:00:48.279299 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.279278    2566 scope.go:117] "RemoveContainer" containerID="9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7"
Jul 24 21:00:48.279649 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:00:48.279629    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7\": container with ID starting with 9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7 not found: ID does not exist" containerID="9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7"
Jul 24 21:00:48.279707 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.279660    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7"} err="failed to get container status \"9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7\": rpc error: code = NotFound desc = could not find container \"9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7\": container with ID starting with 9e2ede67ffe566596efd53135831a9306e861f3451902952d3938bacb8f1e1e7 not found: ID does not exist"
Jul 24 21:00:48.291541 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.291508    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"]
Jul 24 21:00:48.294728 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:48.294710    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-3f62e6/sub-test-model-3f62e6-kserve-687fdb95bc-wlnws"]
Jul 24 21:00:49.698404 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:00:49.695570    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9d02ac4-c855-4c36-ad77-7ad27b4e571b" path="/var/lib/kubelet/pods/f9d02ac4-c855-4c36-ad77-7ad27b4e571b/volumes"
Jul 24 21:01:13.002129 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.002091    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"]
Jul 24 21:01:13.002580 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.002474    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d02ac4-c855-4c36-ad77-7ad27b4e571b" containerName="main"
Jul 24 21:01:13.002580 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.002485    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d02ac4-c855-4c36-ad77-7ad27b4e571b" containerName="main"
Jul 24 21:01:13.002580 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.002499    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24bbe9c0-8eb4-41f9-a872-2a9b64861fed" containerName="cleanup"
Jul 24 21:01:13.002580 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.002504    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="24bbe9c0-8eb4-41f9-a872-2a9b64861fed" containerName="cleanup"
Jul 24 21:01:13.002722 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.002589    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d02ac4-c855-4c36-ad77-7ad27b4e571b" containerName="main"
Jul 24 21:01:13.002722 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.002597    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="24bbe9c0-8eb4-41f9-a872-2a9b64861fed" containerName="cleanup"
Jul 24 21:01:13.005320 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.005302    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.007885 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.007861    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-449ecf\"/\"openshift-service-ca.crt\""
Jul 24 21:01:13.008033 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.008008    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-449ecf\"/\"rate-test-model-449ecf-kserve-self-signed-certs\""
Jul 24 21:01:13.009613 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.009596    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-449ecf\"/\"default-dockercfg-zb9rm\""
Jul 24 21:01:13.009667 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.009657    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-449ecf\"/\"kube-root-ca.crt\""
Jul 24 21:01:13.014587 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.014562    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"]
Jul 24 21:01:13.151799 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.151761    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-dshm\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.151983 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.151825    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d536f49-d895-4803-bc93-02a2b1bf998c-tls-certs\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.151983 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.151847    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fcljx\" (UniqueName: \"kubernetes.io/projected/1d536f49-d895-4803-bc93-02a2b1bf998c-kube-api-access-fcljx\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.151983 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.151868    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-model-cache\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.151983 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.151899    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-home\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.151983 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.151941    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-tmp-dir\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253102 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.252995    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-dshm\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253102 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253062    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d536f49-d895-4803-bc93-02a2b1bf998c-tls-certs\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253102 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253095    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fcljx\" (UniqueName: \"kubernetes.io/projected/1d536f49-d895-4803-bc93-02a2b1bf998c-kube-api-access-fcljx\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253413 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253126    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-model-cache\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253413 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253168    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-home\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253413 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253353    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-tmp-dir\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253593 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253570    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-model-cache\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253637 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253595    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-home\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.253789 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.253767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-tmp-dir\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.255430 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.255400    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-dshm\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.255700 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.255684    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d536f49-d895-4803-bc93-02a2b1bf998c-tls-certs\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.262104 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.262068    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fcljx\" (UniqueName: \"kubernetes.io/projected/1d536f49-d895-4803-bc93-02a2b1bf998c-kube-api-access-fcljx\") pod \"rate-test-model-449ecf-kserve-65677ddf5f-b7bdh\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") " pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.316117 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.316071    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:13.458876 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:13.458839    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"]
Jul 24 21:01:13.461101 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:01:13.461054    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1d536f49_d895_4803_bc93_02a2b1bf998c.slice/crio-f481273de24a49e0e7eff61c6a13eb23e9d0e8539f69b1b6f3bc02e336eb5ffa WatchSource:0}: Error finding container f481273de24a49e0e7eff61c6a13eb23e9d0e8539f69b1b6f3bc02e336eb5ffa: Status 404 returned error can't find the container with id f481273de24a49e0e7eff61c6a13eb23e9d0e8539f69b1b6f3bc02e336eb5ffa
Jul 24 21:01:14.397747 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:14.397711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh" event={"ID":"1d536f49-d895-4803-bc93-02a2b1bf998c","Type":"ContainerStarted","Data":"35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f"}
Jul 24 21:01:14.397747 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:14.397754    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh" event={"ID":"1d536f49-d895-4803-bc93-02a2b1bf998c","Type":"ContainerStarted","Data":"f481273de24a49e0e7eff61c6a13eb23e9d0e8539f69b1b6f3bc02e336eb5ffa"}
Jul 24 21:01:14.414418 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:14.414367    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh" podStartSLOduration=2.414352727 podStartE2EDuration="2.414352727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:01:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:01:14.412815782 +0000 UTC m=+2749.338954687" watchObservedRunningTime="2026-07-24 21:01:14.414352727 +0000 UTC m=+2749.340491631"
Jul 24 21:01:23.317128 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:23.317094    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:23.317632 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:23.317146    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:23.329418 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:23.329393    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:01:23.453212 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:01:23.453179    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:02:33.000874 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:33.000840    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"]
Jul 24 21:02:46.806004 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:46.805963    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:46.809802 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:46.809777    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:46.812475 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:46.812450    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-5zkhj\""
Jul 24 21:02:46.813496 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:46.813474    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:46.871158 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:46.871124    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsv2m\" (UniqueName: \"kubernetes.io/projected/952437ea-6451-41fc-bdc9-3e7f95b50d3b-kube-api-access-tsv2m\") pod \"test-curl-7742-139855873021536\" (UID: \"952437ea-6451-41fc-bdc9-3e7f95b50d3b\") " pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:46.972159 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:46.972120    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsv2m\" (UniqueName: \"kubernetes.io/projected/952437ea-6451-41fc-bdc9-3e7f95b50d3b-kube-api-access-tsv2m\") pod \"test-curl-7742-139855873021536\" (UID: \"952437ea-6451-41fc-bdc9-3e7f95b50d3b\") " pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:46.981393 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:46.981370    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsv2m\" (UniqueName: \"kubernetes.io/projected/952437ea-6451-41fc-bdc9-3e7f95b50d3b-kube-api-access-tsv2m\") pod \"test-curl-7742-139855873021536\" (UID: \"952437ea-6451-41fc-bdc9-3e7f95b50d3b\") " pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:47.120417 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:47.120390    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:47.257258 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:47.257225    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:47.260993 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:02:47.260959    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod952437ea_6451_41fc_bdc9_3e7f95b50d3b.slice/crio-415058bbb49493fc76c1ebe83d36919e0525e5f5994b1400786f89d343c5f2a5 WatchSource:0}: Error finding container 415058bbb49493fc76c1ebe83d36919e0525e5f5994b1400786f89d343c5f2a5: Status 404 returned error can't find the container with id 415058bbb49493fc76c1ebe83d36919e0525e5f5994b1400786f89d343c5f2a5
Jul 24 21:02:47.262388 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:47.262360    2566 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Jul 24 21:02:47.861626 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:47.861582    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139855873021536" event={"ID":"952437ea-6451-41fc-bdc9-3e7f95b50d3b","Type":"ContainerStarted","Data":"415058bbb49493fc76c1ebe83d36919e0525e5f5994b1400786f89d343c5f2a5"}
Jul 24 21:02:48.020216 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.020152    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh" podUID="1d536f49-d895-4803-bc93-02a2b1bf998c" containerName="main" containerID="cri-o://35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f" gracePeriod=45
Jul 24 21:02:48.274897 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.274876    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:02:48.387382 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387290    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-model-cache\") pod \"1d536f49-d895-4803-bc93-02a2b1bf998c\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") "
Jul 24 21:02:48.387382 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387363    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-home\") pod \"1d536f49-d895-4803-bc93-02a2b1bf998c\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") "
Jul 24 21:02:48.387655 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387460    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d536f49-d895-4803-bc93-02a2b1bf998c-tls-certs\") pod \"1d536f49-d895-4803-bc93-02a2b1bf998c\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") "
Jul 24 21:02:48.387655 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387506    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-dshm\") pod \"1d536f49-d895-4803-bc93-02a2b1bf998c\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") "
Jul 24 21:02:48.387655 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387556    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-tmp-dir\") pod \"1d536f49-d895-4803-bc93-02a2b1bf998c\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") "
Jul 24 21:02:48.387816 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387658    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fcljx\" (UniqueName: \"kubernetes.io/projected/1d536f49-d895-4803-bc93-02a2b1bf998c-kube-api-access-fcljx\") pod \"1d536f49-d895-4803-bc93-02a2b1bf998c\" (UID: \"1d536f49-d895-4803-bc93-02a2b1bf998c\") "
Jul 24 21:02:48.387816 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387659    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-model-cache" (OuterVolumeSpecName: "model-cache") pod "1d536f49-d895-4803-bc93-02a2b1bf998c" (UID: "1d536f49-d895-4803-bc93-02a2b1bf998c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:02:48.387816 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387679    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-home" (OuterVolumeSpecName: "home") pod "1d536f49-d895-4803-bc93-02a2b1bf998c" (UID: "1d536f49-d895-4803-bc93-02a2b1bf998c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:02:48.387816 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387805    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1d536f49-d895-4803-bc93-02a2b1bf998c" (UID: "1d536f49-d895-4803-bc93-02a2b1bf998c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:02:48.388007 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.387986    2566 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-model-cache\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:48.388007 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.388004    2566 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-home\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:48.388072 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.388012    2566 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-tmp-dir\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:48.389799 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.389775    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d536f49-d895-4803-bc93-02a2b1bf998c-kube-api-access-fcljx" (OuterVolumeSpecName: "kube-api-access-fcljx") pod "1d536f49-d895-4803-bc93-02a2b1bf998c" (UID: "1d536f49-d895-4803-bc93-02a2b1bf998c"). InnerVolumeSpecName "kube-api-access-fcljx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:02:48.389799 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.389789    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d536f49-d895-4803-bc93-02a2b1bf998c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1d536f49-d895-4803-bc93-02a2b1bf998c" (UID: "1d536f49-d895-4803-bc93-02a2b1bf998c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:02:48.389918 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.389825    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-dshm" (OuterVolumeSpecName: "dshm") pod "1d536f49-d895-4803-bc93-02a2b1bf998c" (UID: "1d536f49-d895-4803-bc93-02a2b1bf998c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:02:48.488851 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.488814    2566 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1d536f49-d895-4803-bc93-02a2b1bf998c-tls-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:48.488851 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.488844    2566 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1d536f49-d895-4803-bc93-02a2b1bf998c-dshm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:48.488851 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.488854    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fcljx\" (UniqueName: \"kubernetes.io/projected/1d536f49-d895-4803-bc93-02a2b1bf998c-kube-api-access-fcljx\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:48.867842 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.867805    2566 generic.go:356] "Generic (PLEG): container finished" podID="952437ea-6451-41fc-bdc9-3e7f95b50d3b" containerID="7c05289b429e5a1047eaea4e2cec73db946095b450095985baa150f87b3db5aa" exitCode=0
Jul 24 21:02:48.868290 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.867893    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139855873021536" event={"ID":"952437ea-6451-41fc-bdc9-3e7f95b50d3b","Type":"ContainerDied","Data":"7c05289b429e5a1047eaea4e2cec73db946095b450095985baa150f87b3db5aa"}
Jul 24 21:02:48.869276 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.869251    2566 generic.go:356] "Generic (PLEG): container finished" podID="1d536f49-d895-4803-bc93-02a2b1bf998c" containerID="35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f" exitCode=0
Jul 24 21:02:48.869361 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.869280    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh" event={"ID":"1d536f49-d895-4803-bc93-02a2b1bf998c","Type":"ContainerDied","Data":"35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f"}
Jul 24 21:02:48.869361 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.869301    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"
Jul 24 21:02:48.869361 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.869321    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh" event={"ID":"1d536f49-d895-4803-bc93-02a2b1bf998c","Type":"ContainerDied","Data":"f481273de24a49e0e7eff61c6a13eb23e9d0e8539f69b1b6f3bc02e336eb5ffa"}
Jul 24 21:02:48.869361 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.869342    2566 scope.go:117] "RemoveContainer" containerID="35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f"
Jul 24 21:02:48.878682 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.878666    2566 scope.go:117] "RemoveContainer" containerID="35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f"
Jul 24 21:02:48.878944 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:02:48.878928    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f\": container with ID starting with 35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f not found: ID does not exist" containerID="35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f"
Jul 24 21:02:48.878987 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.878951    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f"} err="failed to get container status \"35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f\": rpc error: code = NotFound desc = could not find container \"35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f\": container with ID starting with 35b52bcf9ee983db1f5eb1d15ecf1c145670a741e5943492469ea35eb9fbea8f not found: ID does not exist"
Jul 24 21:02:48.895047 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.895020    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"]
Jul 24 21:02:48.902195 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:48.898517    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-449ecf/rate-test-model-449ecf-kserve-65677ddf5f-b7bdh"]
Jul 24 21:02:49.693050 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:49.693014    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1d536f49-d895-4803-bc93-02a2b1bf998c" path="/var/lib/kubelet/pods/1d536f49-d895-4803-bc93-02a2b1bf998c/volumes"
Jul 24 21:02:50.009839 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.009812    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:50.043165 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.043142    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139855873021536_952437ea-6451-41fc-bdc9-3e7f95b50d3b/test-curl-7742-139855873021536/0.log"
Jul 24 21:02:50.067759 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.067723    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:50.073290 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.073262    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:50.104160 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.104131    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tsv2m\" (UniqueName: \"kubernetes.io/projected/952437ea-6451-41fc-bdc9-3e7f95b50d3b-kube-api-access-tsv2m\") pod \"952437ea-6451-41fc-bdc9-3e7f95b50d3b\" (UID: \"952437ea-6451-41fc-bdc9-3e7f95b50d3b\") "
Jul 24 21:02:50.106171 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.106148    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/952437ea-6451-41fc-bdc9-3e7f95b50d3b-kube-api-access-tsv2m" (OuterVolumeSpecName: "kube-api-access-tsv2m") pod "952437ea-6451-41fc-bdc9-3e7f95b50d3b" (UID: "952437ea-6451-41fc-bdc9-3e7f95b50d3b"). InnerVolumeSpecName "kube-api-access-tsv2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:02:50.190492 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.190459    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:50.190890 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.190875    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="952437ea-6451-41fc-bdc9-3e7f95b50d3b" containerName="test-curl-7742-139855873021536"
Jul 24 21:02:50.190937 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.190891    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="952437ea-6451-41fc-bdc9-3e7f95b50d3b" containerName="test-curl-7742-139855873021536"
Jul 24 21:02:50.190937 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.190911    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1d536f49-d895-4803-bc93-02a2b1bf998c" containerName="main"
Jul 24 21:02:50.190937 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.190921    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="1d536f49-d895-4803-bc93-02a2b1bf998c" containerName="main"
Jul 24 21:02:50.191035 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.190975    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="1d536f49-d895-4803-bc93-02a2b1bf998c" containerName="main"
Jul 24 21:02:50.191035 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.190985    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="952437ea-6451-41fc-bdc9-3e7f95b50d3b" containerName="test-curl-7742-139855873021536"
Jul 24 21:02:50.193978 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.193962    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:50.200455 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.200431    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:50.205317 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.205294    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsv2m\" (UniqueName: \"kubernetes.io/projected/952437ea-6451-41fc-bdc9-3e7f95b50d3b-kube-api-access-tsv2m\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:50.306842 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.306749    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6txcg\" (UniqueName: \"kubernetes.io/projected/017ac1b3-ab46-45dd-86ff-45b05028c309-kube-api-access-6txcg\") pod \"test-curl-7742-139855873021536\" (UID: \"017ac1b3-ab46-45dd-86ff-45b05028c309\") " pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:50.407637 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.407600    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6txcg\" (UniqueName: \"kubernetes.io/projected/017ac1b3-ab46-45dd-86ff-45b05028c309-kube-api-access-6txcg\") pod \"test-curl-7742-139855873021536\" (UID: \"017ac1b3-ab46-45dd-86ff-45b05028c309\") " pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:50.415927 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.415903    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6txcg\" (UniqueName: \"kubernetes.io/projected/017ac1b3-ab46-45dd-86ff-45b05028c309-kube-api-access-6txcg\") pod \"test-curl-7742-139855873021536\" (UID: \"017ac1b3-ab46-45dd-86ff-45b05028c309\") " pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:50.504186 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.504154    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:50.631756 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.631729    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:50.634219 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:02:50.634182    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod017ac1b3_ab46_45dd_86ff_45b05028c309.slice/crio-2442ec0aa64d53f62c6052729c00c91f455092e82a778ea864a26ec4d3873457 WatchSource:0}: Error finding container 2442ec0aa64d53f62c6052729c00c91f455092e82a778ea864a26ec4d3873457: Status 404 returned error can't find the container with id 2442ec0aa64d53f62c6052729c00c91f455092e82a778ea864a26ec4d3873457
Jul 24 21:02:50.880991 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.880963    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:50.881155 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.881009    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415058bbb49493fc76c1ebe83d36919e0525e5f5994b1400786f89d343c5f2a5"
Jul 24 21:02:50.882193 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.882163    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139855873021536" event={"ID":"017ac1b3-ab46-45dd-86ff-45b05028c309","Type":"ContainerStarted","Data":"2442ec0aa64d53f62c6052729c00c91f455092e82a778ea864a26ec4d3873457"}
Jul 24 21:02:50.893421 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:50.893387    2566 status_manager.go:1027] "Pod was deleted and then recreated, skipping status update" pod="opendatahub/test-curl-7742-139855873021536" oldPodUID="952437ea-6451-41fc-bdc9-3e7f95b50d3b" podUID="017ac1b3-ab46-45dd-86ff-45b05028c309"
Jul 24 21:02:51.692651 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:51.692617    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="952437ea-6451-41fc-bdc9-3e7f95b50d3b" path="/var/lib/kubelet/pods/952437ea-6451-41fc-bdc9-3e7f95b50d3b/volumes"
Jul 24 21:02:51.888889 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:51.888853    2566 generic.go:356] "Generic (PLEG): container finished" podID="017ac1b3-ab46-45dd-86ff-45b05028c309" containerID="2880e86eeda0565c580bdcd6e1c9165db0274158b261020fbfeec9ed2e74df8d" exitCode=0
Jul 24 21:02:51.889060 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:51.888944    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7742-139855873021536" event={"ID":"017ac1b3-ab46-45dd-86ff-45b05028c309","Type":"ContainerDied","Data":"2880e86eeda0565c580bdcd6e1c9165db0274158b261020fbfeec9ed2e74df8d"}
Jul 24 21:02:53.029044 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.029021    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:53.048194 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.048172    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7742-139855873021536_017ac1b3-ab46-45dd-86ff-45b05028c309/test-curl-7742-139855873021536/0.log"
Jul 24 21:02:53.070719 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.070691    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:53.074238 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.074214    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7742-139855873021536"]
Jul 24 21:02:53.133228 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.133195    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6txcg\" (UniqueName: \"kubernetes.io/projected/017ac1b3-ab46-45dd-86ff-45b05028c309-kube-api-access-6txcg\") pod \"017ac1b3-ab46-45dd-86ff-45b05028c309\" (UID: \"017ac1b3-ab46-45dd-86ff-45b05028c309\") "
Jul 24 21:02:53.135279 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.135248    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/017ac1b3-ab46-45dd-86ff-45b05028c309-kube-api-access-6txcg" (OuterVolumeSpecName: "kube-api-access-6txcg") pod "017ac1b3-ab46-45dd-86ff-45b05028c309" (UID: "017ac1b3-ab46-45dd-86ff-45b05028c309"). InnerVolumeSpecName "kube-api-access-6txcg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:02:53.234398 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.234293    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6txcg\" (UniqueName: \"kubernetes.io/projected/017ac1b3-ab46-45dd-86ff-45b05028c309-kube-api-access-6txcg\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:02:53.699040 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.699005    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="017ac1b3-ab46-45dd-86ff-45b05028c309" path="/var/lib/kubelet/pods/017ac1b3-ab46-45dd-86ff-45b05028c309/volumes"
Jul 24 21:02:53.901439 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.901411    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7742-139855873021536"
Jul 24 21:02:53.902482 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:53.901440    2566 scope.go:117] "RemoveContainer" containerID="2880e86eeda0565c580bdcd6e1c9165db0274158b261020fbfeec9ed2e74df8d"
Jul 24 21:02:57.592474 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.592440    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"]
Jul 24 21:02:57.592899 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.592856    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="017ac1b3-ab46-45dd-86ff-45b05028c309" containerName="test-curl-7742-139855873021536"
Jul 24 21:02:57.592899 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.592869    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="017ac1b3-ab46-45dd-86ff-45b05028c309" containerName="test-curl-7742-139855873021536"
Jul 24 21:02:57.592983 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.592930    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="017ac1b3-ab46-45dd-86ff-45b05028c309" containerName="test-curl-7742-139855873021536"
Jul 24 21:02:57.597323 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.597301    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.600031 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.600008    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-a-787669-serving-cert\""
Jul 24 21:02:57.607413 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.607389    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"]
Jul 24 21:02:57.678387 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.678353    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xhzx\" (UniqueName: \"kubernetes.io/projected/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-kube-api-access-9xhzx\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.678387 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.678392    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.678695 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.678510    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.779810 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.779771    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.780005 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.779882    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xhzx\" (UniqueName: \"kubernetes.io/projected/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-kube-api-access-9xhzx\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.780005 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.779908    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.780005 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:02:57.779995    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-serving-cert: secret "maas-api-e2e-ipp-a-787669-serving-cert" not found
Jul 24 21:02:57.780185 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:02:57.780092    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls podName:0a9411d9-8957-4c86-ba9b-4a7e007bde0e nodeName:}" failed. No retries permitted until 2026-07-24 21:02:58.280069492 +0000 UTC m=+2853.206208380 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls") pod "maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf" (UID: "0a9411d9-8957-4c86-ba9b-4a7e007bde0e") : secret "maas-api-e2e-ipp-a-787669-serving-cert" not found
Jul 24 21:02:57.780590 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.780571    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-openshift-service-ca\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:57.788863 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:57.788841    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xhzx\" (UniqueName: \"kubernetes.io/projected/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-kube-api-access-9xhzx\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:58.283074 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.283032    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"]
Jul 24 21:02:58.285316 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.285276    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:58.287026 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.287001    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.288794 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.288767    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls\") pod \"maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:58.289843 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.289821    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-787669\""
Jul 24 21:02:58.289937 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.289821    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-787669-dockercfg-tffsq\""
Jul 24 21:02:58.297999 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.297972    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"]
Jul 24 21:02:58.386147 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.386109    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r9jmm\" (UniqueName: \"kubernetes.io/projected/45017c38-2ef7-45bf-8ab1-1b93366a9f21-kube-api-access-r9jmm\") pod \"payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.386353 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.386168    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45017c38-2ef7-45bf-8ab1-1b93366a9f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.487145 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.487104    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45017c38-2ef7-45bf-8ab1-1b93366a9f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.487366 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.487264    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r9jmm\" (UniqueName: \"kubernetes.io/projected/45017c38-2ef7-45bf-8ab1-1b93366a9f21-kube-api-access-r9jmm\") pod \"payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.487957 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.487934    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45017c38-2ef7-45bf-8ab1-1b93366a9f21-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.494833 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.494811    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r9jmm\" (UniqueName: \"kubernetes.io/projected/45017c38-2ef7-45bf-8ab1-1b93366a9f21-kube-api-access-r9jmm\") pod \"payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.508683 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.508662    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:02:58.607863 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.607829    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.647239 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.646933    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"]
Jul 24 21:02:58.650063 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:02:58.650029    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0a9411d9_8957_4c86_ba9b_4a7e007bde0e.slice/crio-8b5eb1aace78f54a765c06757fa9b35e89f4f7d697fdef5ed4abee824067c672 WatchSource:0}: Error finding container 8b5eb1aace78f54a765c06757fa9b35e89f4f7d697fdef5ed4abee824067c672: Status 404 returned error can't find the container with id 8b5eb1aace78f54a765c06757fa9b35e89f4f7d697fdef5ed4abee824067c672
Jul 24 21:02:58.786707 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.786677    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"]
Jul 24 21:02:58.787597 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:02:58.787572    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45017c38_2ef7_45bf_8ab1_1b93366a9f21.slice/crio-2a4a6084b40ad336ede2bfd65ed4bfa5978c96797864966bc1b7bc724503facd WatchSource:0}: Error finding container 2a4a6084b40ad336ede2bfd65ed4bfa5978c96797864966bc1b7bc724503facd: Status 404 returned error can't find the container with id 2a4a6084b40ad336ede2bfd65ed4bfa5978c96797864966bc1b7bc724503facd
Jul 24 21:02:58.931653 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.931612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr" event={"ID":"45017c38-2ef7-45bf-8ab1-1b93366a9f21","Type":"ContainerStarted","Data":"155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9"}
Jul 24 21:02:58.931854 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.931711    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr" event={"ID":"45017c38-2ef7-45bf-8ab1-1b93366a9f21","Type":"ContainerStarted","Data":"2a4a6084b40ad336ede2bfd65ed4bfa5978c96797864966bc1b7bc724503facd"}
Jul 24 21:02:58.931854 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.931744    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:02:58.932960 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.932938    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf" event={"ID":"0a9411d9-8957-4c86-ba9b-4a7e007bde0e","Type":"ContainerStarted","Data":"8b5eb1aace78f54a765c06757fa9b35e89f4f7d697fdef5ed4abee824067c672"}
Jul 24 21:02:58.952042 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:58.951980    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr" podStartSLOduration=0.951958467 podStartE2EDuration="951.958467ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:02:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:02:58.949053402 +0000 UTC m=+2853.875192341" watchObservedRunningTime="2026-07-24 21:02:58.951958467 +0000 UTC m=+2853.878097373"
Jul 24 21:02:59.940383 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:59.940344    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf" event={"ID":"0a9411d9-8957-4c86-ba9b-4a7e007bde0e","Type":"ContainerStarted","Data":"81880abe4abdf91cab97cf16c26f16dd28b1059933491713f3c59763f5d77fee"}
Jul 24 21:02:59.956888 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:02:59.956839    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf" podStartSLOduration=2.343055527 podStartE2EDuration="2.956824528s" totalImagesPullingTime="613.769001ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:02:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:02:59.956296914 +0000 UTC m=+2854.882435839" watchObservedRunningTime="2026-07-24 21:02:59.956824528 +0000 UTC m=+2854.882963466"
Jul 24 21:03:00.945588 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:00.945549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:03:06.956817 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:06.956788    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:03:09.943308 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:09.943274    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:03:16.602585 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.602546    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"]
Jul 24 21:03:16.606687 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.606659    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.609111 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.609082    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-b-5df102-serving-cert\""
Jul 24 21:03:16.619164 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.619097    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"]
Jul 24 21:03:16.661467 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.661422    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz9fn\" (UniqueName: \"kubernetes.io/projected/91d58022-e123-4629-b91d-46fb8f77d121-kube-api-access-jz9fn\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.661651 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.661544    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.661651 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.661634    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d58022-e123-4629-b91d-46fb8f77d121-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.762070 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.762033    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d58022-e123-4629-b91d-46fb8f77d121-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.762070 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.762075    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz9fn\" (UniqueName: \"kubernetes.io/projected/91d58022-e123-4629-b91d-46fb8f77d121-kube-api-access-jz9fn\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.762291 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.762149    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.762291 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:03:16.762259    2566 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-serving-cert: secret "maas-api-e2e-ipp-b-5df102-serving-cert" not found
Jul 24 21:03:16.762363 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:03:16.762329    2566 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls podName:91d58022-e123-4629-b91d-46fb8f77d121 nodeName:}" failed. No retries permitted until 2026-07-24 21:03:17.262306373 +0000 UTC m=+2872.188445273 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls") pod "maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5" (UID: "91d58022-e123-4629-b91d-46fb8f77d121") : secret "maas-api-e2e-ipp-b-5df102-serving-cert" not found
Jul 24 21:03:16.762874 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.762857    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d58022-e123-4629-b91d-46fb8f77d121-openshift-service-ca\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:16.771026 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:16.771005    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz9fn\" (UniqueName: \"kubernetes.io/projected/91d58022-e123-4629-b91d-46fb8f77d121-kube-api-access-jz9fn\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:17.267705 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:17.267661    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:17.270157 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:17.270126    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls\") pod \"maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:17.521636 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:17.521546    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:17.663132 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:17.663101    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"]
Jul 24 21:03:17.664168 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:03:17.664139    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91d58022_e123_4629_b91d_46fb8f77d121.slice/crio-024f341dcca5ec557b684247be556e05b4c450404ec19109d5507f31985fca3f WatchSource:0}: Error finding container 024f341dcca5ec557b684247be556e05b4c450404ec19109d5507f31985fca3f: Status 404 returned error can't find the container with id 024f341dcca5ec557b684247be556e05b4c450404ec19109d5507f31985fca3f
Jul 24 21:03:18.034831 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:18.034795    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5" event={"ID":"91d58022-e123-4629-b91d-46fb8f77d121","Type":"ContainerStarted","Data":"024f341dcca5ec557b684247be556e05b4c450404ec19109d5507f31985fca3f"}
Jul 24 21:03:19.041357 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:19.041317    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5" event={"ID":"91d58022-e123-4629-b91d-46fb8f77d121","Type":"ContainerStarted","Data":"46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2"}
Jul 24 21:03:19.041848 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:19.041542    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:03:19.058545 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:19.058477    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5" podStartSLOduration=2.484358714 podStartE2EDuration="3.058462876s" totalImagesPullingTime="574.104162ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:03:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:03:19.057806492 +0000 UTC m=+2873.983945420" watchObservedRunningTime="2026-07-24 21:03:19.058462876 +0000 UTC m=+2873.984601781"
Jul 24 21:03:25.051811 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:03:25.051786    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:04:45.597234 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:45.597205    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr_45017c38-2ef7-45bf-8ab1-1b93366a9f21/payload-processing/0.log"
Jul 24 21:04:50.035050 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.034963    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"]
Jul 24 21:04:50.038610 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.038588    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.041106 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.041076    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ipp-cleanup-fd3ebf-serving-cert\""
Jul 24 21:04:50.048123 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.048095    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"]
Jul 24 21:04:50.174298 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.174263    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb831d7c-3a57-44f0-bbd2-39db30dc1775-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.174298 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.174300    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/fb831d7c-3a57-44f0-bbd2-39db30dc1775-kube-api-access-hz8gh\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.174553 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.174333    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb831d7c-3a57-44f0-bbd2-39db30dc1775-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.274920 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.274881    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb831d7c-3a57-44f0-bbd2-39db30dc1775-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.274920 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.274920    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/fb831d7c-3a57-44f0-bbd2-39db30dc1775-kube-api-access-hz8gh\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.275189 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.274950    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb831d7c-3a57-44f0-bbd2-39db30dc1775-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.275551 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.275507    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb831d7c-3a57-44f0-bbd2-39db30dc1775-openshift-service-ca\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.277391 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.277373    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb831d7c-3a57-44f0-bbd2-39db30dc1775-maas-api-tls\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.287171 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.287115    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/fb831d7c-3a57-44f0-bbd2-39db30dc1775-kube-api-access-hz8gh\") pod \"maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") " pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.351442 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.351409    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:50.502230 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:50.502196    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"]
Jul 24 21:04:50.504218 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:04:50.504181    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb831d7c_3a57_44f0_bbd2_39db30dc1775.slice/crio-b45cc05f9135c23ce6de95b6632c36db4b859ca2c3ae40f86dfb2ff91d3c5b4b WatchSource:0}: Error finding container b45cc05f9135c23ce6de95b6632c36db4b859ca2c3ae40f86dfb2ff91d3c5b4b: Status 404 returned error can't find the container with id b45cc05f9135c23ce6de95b6632c36db4b859ca2c3ae40f86dfb2ff91d3c5b4b
Jul 24 21:04:51.513857 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:51.513823    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89" event={"ID":"fb831d7c-3a57-44f0-bbd2-39db30dc1775","Type":"ContainerStarted","Data":"82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20"}
Jul 24 21:04:51.514260 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:51.513877    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89" event={"ID":"fb831d7c-3a57-44f0-bbd2-39db30dc1775","Type":"ContainerStarted","Data":"b45cc05f9135c23ce6de95b6632c36db4b859ca2c3ae40f86dfb2ff91d3c5b4b"}
Jul 24 21:04:51.514260 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:51.513950    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:04:51.530660 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:51.530605    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89" podStartSLOduration=1.128680051 podStartE2EDuration="1.530591177s" totalImagesPullingTime="401.911126ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:04:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:04:51.529721001 +0000 UTC m=+2966.455859917" watchObservedRunningTime="2026-07-24 21:04:51.530591177 +0000 UTC m=+2966.456730076"
Jul 24 21:04:57.524970 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:04:57.524937    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:05:05.384229 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.384190    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"]
Jul 24 21:05:05.387738 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.387723    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"
Jul 24 21:05:05.392681 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.392663    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 21:05:05.401071 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.401045    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"]
Jul 24 21:05:05.419871 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.419842    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023116e5-5a09-41d0-b975-6bff0debd642-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z\" (UID: \"023116e5-5a09-41d0-b975-6bff0debd642\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"
Jul 24 21:05:05.521316 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.521263    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023116e5-5a09-41d0-b975-6bff0debd642-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z\" (UID: \"023116e5-5a09-41d0-b975-6bff0debd642\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"
Jul 24 21:05:05.521923 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.521902    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023116e5-5a09-41d0-b975-6bff0debd642-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z\" (UID: \"023116e5-5a09-41d0-b975-6bff0debd642\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"
Jul 24 21:05:05.698461 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.698384    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"
Jul 24 21:05:05.841860 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:05.841834    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"]
Jul 24 21:05:05.844837 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:05:05.844803    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod023116e5_5a09_41d0_b975_6bff0debd642.slice/crio-3273c9ba9045ad7894e1753196afb1bf36b7be89265569423c7b50965a067505 WatchSource:0}: Error finding container 3273c9ba9045ad7894e1753196afb1bf36b7be89265569423c7b50965a067505: Status 404 returned error can't find the container with id 3273c9ba9045ad7894e1753196afb1bf36b7be89265569423c7b50965a067505
Jul 24 21:05:06.592339 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:06.592303    2566 generic.go:356] "Generic (PLEG): container finished" podID="023116e5-5a09-41d0-b975-6bff0debd642" containerID="e6ac9cfa2bfe901646de224e1df55ae89ed726224adba5a2f3b4bcda9b1f8d59" exitCode=0
Jul 24 21:05:06.592745 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:06.592363    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z" event={"ID":"023116e5-5a09-41d0-b975-6bff0debd642","Type":"ContainerDied","Data":"e6ac9cfa2bfe901646de224e1df55ae89ed726224adba5a2f3b4bcda9b1f8d59"}
Jul 24 21:05:06.592745 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:06.592446    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z" event={"ID":"023116e5-5a09-41d0-b975-6bff0debd642","Type":"ContainerStarted","Data":"3273c9ba9045ad7894e1753196afb1bf36b7be89265569423c7b50965a067505"}
Jul 24 21:05:07.733263 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:07.733238    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"
Jul 24 21:05:07.844486 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:07.844447    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023116e5-5a09-41d0-b975-6bff0debd642-maas-api-service-ca\") pod \"023116e5-5a09-41d0-b975-6bff0debd642\" (UID: \"023116e5-5a09-41d0-b975-6bff0debd642\") "
Jul 24 21:05:07.844846 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:07.844822    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/023116e5-5a09-41d0-b975-6bff0debd642-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "023116e5-5a09-41d0-b975-6bff0debd642" (UID: "023116e5-5a09-41d0-b975-6bff0debd642"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:05:07.946300 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:07.946223    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/023116e5-5a09-41d0-b975-6bff0debd642-maas-api-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:08.605289 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:08.605254    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"
Jul 24 21:05:08.605289 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:08.605285    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z" event={"ID":"023116e5-5a09-41d0-b975-6bff0debd642","Type":"ContainerDied","Data":"3273c9ba9045ad7894e1753196afb1bf36b7be89265569423c7b50965a067505"}
Jul 24 21:05:08.605519 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:08.605337    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3273c9ba9045ad7894e1753196afb1bf36b7be89265569423c7b50965a067505"
Jul 24 21:05:16.639583 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:16.639542    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"]
Jul 24 21:05:16.640207 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:16.640123    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89" podUID="fb831d7c-3a57-44f0-bbd2-39db30dc1775" containerName="maas-api" containerID="cri-o://82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20" gracePeriod=30
Jul 24 21:05:16.917092 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:16.917060    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:05:17.042149 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.042119    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb831d7c-3a57-44f0-bbd2-39db30dc1775-openshift-service-ca\") pod \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") "
Jul 24 21:05:17.042321 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.042187    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb831d7c-3a57-44f0-bbd2-39db30dc1775-maas-api-tls\") pod \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") "
Jul 24 21:05:17.042321 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.042222    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/fb831d7c-3a57-44f0-bbd2-39db30dc1775-kube-api-access-hz8gh\") pod \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\" (UID: \"fb831d7c-3a57-44f0-bbd2-39db30dc1775\") "
Jul 24 21:05:17.042596 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.042563    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb831d7c-3a57-44f0-bbd2-39db30dc1775-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "fb831d7c-3a57-44f0-bbd2-39db30dc1775" (UID: "fb831d7c-3a57-44f0-bbd2-39db30dc1775"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:05:17.044296 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.044265    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb831d7c-3a57-44f0-bbd2-39db30dc1775-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "fb831d7c-3a57-44f0-bbd2-39db30dc1775" (UID: "fb831d7c-3a57-44f0-bbd2-39db30dc1775"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:05:17.044431 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.044407    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb831d7c-3a57-44f0-bbd2-39db30dc1775-kube-api-access-hz8gh" (OuterVolumeSpecName: "kube-api-access-hz8gh") pod "fb831d7c-3a57-44f0-bbd2-39db30dc1775" (UID: "fb831d7c-3a57-44f0-bbd2-39db30dc1775"). InnerVolumeSpecName "kube-api-access-hz8gh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:05:17.143556 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.143503    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/fb831d7c-3a57-44f0-bbd2-39db30dc1775-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:17.143556 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.143548    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8gh\" (UniqueName: \"kubernetes.io/projected/fb831d7c-3a57-44f0-bbd2-39db30dc1775-kube-api-access-hz8gh\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:17.143556 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.143559    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/fb831d7c-3a57-44f0-bbd2-39db30dc1775-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:17.655415 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.655374    2566 generic.go:356] "Generic (PLEG): container finished" podID="fb831d7c-3a57-44f0-bbd2-39db30dc1775" containerID="82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20" exitCode=0
Jul 24 21:05:17.655415 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.655418    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"
Jul 24 21:05:17.656006 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.655462    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89" event={"ID":"fb831d7c-3a57-44f0-bbd2-39db30dc1775","Type":"ContainerDied","Data":"82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20"}
Jul 24 21:05:17.656006 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.655515    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89" event={"ID":"fb831d7c-3a57-44f0-bbd2-39db30dc1775","Type":"ContainerDied","Data":"b45cc05f9135c23ce6de95b6632c36db4b859ca2c3ae40f86dfb2ff91d3c5b4b"}
Jul 24 21:05:17.656006 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.655553    2566 scope.go:117] "RemoveContainer" containerID="82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20"
Jul 24 21:05:17.671610 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.671586    2566 scope.go:117] "RemoveContainer" containerID="82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20"
Jul 24 21:05:17.671977 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:05:17.671945    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20\": container with ID starting with 82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20 not found: ID does not exist" containerID="82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20"
Jul 24 21:05:17.672078 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.671989    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20"} err="failed to get container status \"82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20\": rpc error: code = NotFound desc = could not find container \"82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20\": container with ID starting with 82a2595bf467431859e2241ce2fe64e3c9729c743600f116e51cd41813991a20 not found: ID does not exist"
Jul 24 21:05:17.683448 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.683420    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"]
Jul 24 21:05:17.693744 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:17.693719    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-cleanup-fd3ebf-d6b8cbc8f-5mc89"]
Jul 24 21:05:19.692288 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:19.692253    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fb831d7c-3a57-44f0-bbd2-39db30dc1775" path="/var/lib/kubelet/pods/fb831d7c-3a57-44f0-bbd2-39db30dc1775/volumes"
Jul 24 21:05:20.477121 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:20.477081    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"]
Jul 24 21:05:20.487746 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:20.487707    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-fd3ebf-9vr6z"]
Jul 24 21:05:21.693330 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:21.693292    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="023116e5-5a09-41d0-b975-6bff0debd642" path="/var/lib/kubelet/pods/023116e5-5a09-41d0-b975-6bff0debd642/volumes"
Jul 24 21:05:46.599953 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.599891    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"]
Jul 24 21:05:46.600581 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.600146    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf" podUID="0a9411d9-8957-4c86-ba9b-4a7e007bde0e" containerName="maas-api" containerID="cri-o://81880abe4abdf91cab97cf16c26f16dd28b1059933491713f3c59763f5d77fee" gracePeriod=30
Jul 24 21:05:46.691783 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.691742    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"]
Jul 24 21:05:46.692015 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.691992    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr" podUID="45017c38-2ef7-45bf-8ab1-1b93366a9f21" containerName="payload-processing" containerID="cri-o://155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9" gracePeriod=30
Jul 24 21:05:46.806141 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.806103    2566 generic.go:356] "Generic (PLEG): container finished" podID="0a9411d9-8957-4c86-ba9b-4a7e007bde0e" containerID="81880abe4abdf91cab97cf16c26f16dd28b1059933491713f3c59763f5d77fee" exitCode=0
Jul 24 21:05:46.806310 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.806263    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf" event={"ID":"0a9411d9-8957-4c86-ba9b-4a7e007bde0e","Type":"ContainerDied","Data":"81880abe4abdf91cab97cf16c26f16dd28b1059933491713f3c59763f5d77fee"}
Jul 24 21:05:46.955356 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.955310    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:05:46.995009 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:46.994984    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:05:47.040006 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.039973    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xhzx\" (UniqueName: \"kubernetes.io/projected/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-kube-api-access-9xhzx\") pod \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") "
Jul 24 21:05:47.040204 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.040029    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-openshift-service-ca\") pod \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") "
Jul 24 21:05:47.040204 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.040048    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r9jmm\" (UniqueName: \"kubernetes.io/projected/45017c38-2ef7-45bf-8ab1-1b93366a9f21-kube-api-access-r9jmm\") pod \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") "
Jul 24 21:05:47.040204 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.040091    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45017c38-2ef7-45bf-8ab1-1b93366a9f21-plugins-config-volume\") pod \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\" (UID: \"45017c38-2ef7-45bf-8ab1-1b93366a9f21\") "
Jul 24 21:05:47.040204 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.040176    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls\") pod \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\" (UID: \"0a9411d9-8957-4c86-ba9b-4a7e007bde0e\") "
Jul 24 21:05:47.040633 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.040599    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45017c38-2ef7-45bf-8ab1-1b93366a9f21-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "45017c38-2ef7-45bf-8ab1-1b93366a9f21" (UID: "45017c38-2ef7-45bf-8ab1-1b93366a9f21"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:05:47.041103 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.041070    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "0a9411d9-8957-4c86-ba9b-4a7e007bde0e" (UID: "0a9411d9-8957-4c86-ba9b-4a7e007bde0e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:05:47.042340 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.042313    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-kube-api-access-9xhzx" (OuterVolumeSpecName: "kube-api-access-9xhzx") pod "0a9411d9-8957-4c86-ba9b-4a7e007bde0e" (UID: "0a9411d9-8957-4c86-ba9b-4a7e007bde0e"). InnerVolumeSpecName "kube-api-access-9xhzx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:05:47.042555 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.042510    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "0a9411d9-8957-4c86-ba9b-4a7e007bde0e" (UID: "0a9411d9-8957-4c86-ba9b-4a7e007bde0e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:05:47.042696 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.042673    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45017c38-2ef7-45bf-8ab1-1b93366a9f21-kube-api-access-r9jmm" (OuterVolumeSpecName: "kube-api-access-r9jmm") pod "45017c38-2ef7-45bf-8ab1-1b93366a9f21" (UID: "45017c38-2ef7-45bf-8ab1-1b93366a9f21"). InnerVolumeSpecName "kube-api-access-r9jmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:05:47.142114 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.142016    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xhzx\" (UniqueName: \"kubernetes.io/projected/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-kube-api-access-9xhzx\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:47.142114 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.142043    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:47.142114 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.142054    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r9jmm\" (UniqueName: \"kubernetes.io/projected/45017c38-2ef7-45bf-8ab1-1b93366a9f21-kube-api-access-r9jmm\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:47.142114 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.142063    2566 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/45017c38-2ef7-45bf-8ab1-1b93366a9f21-plugins-config-volume\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:47.142114 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.142072    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0a9411d9-8957-4c86-ba9b-4a7e007bde0e-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:05:47.812042 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.812002    2566 generic.go:356] "Generic (PLEG): container finished" podID="45017c38-2ef7-45bf-8ab1-1b93366a9f21" containerID="155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9" exitCode=0
Jul 24 21:05:47.812042 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.812046    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"
Jul 24 21:05:47.812620 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.812083    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr" event={"ID":"45017c38-2ef7-45bf-8ab1-1b93366a9f21","Type":"ContainerDied","Data":"155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9"}
Jul 24 21:05:47.812620 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.812134    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr" event={"ID":"45017c38-2ef7-45bf-8ab1-1b93366a9f21","Type":"ContainerDied","Data":"2a4a6084b40ad336ede2bfd65ed4bfa5978c96797864966bc1b7bc724503facd"}
Jul 24 21:05:47.812620 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.812160    2566 scope.go:117] "RemoveContainer" containerID="155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9"
Jul 24 21:05:47.813486 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.813469    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"
Jul 24 21:05:47.813573 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.813467    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf" event={"ID":"0a9411d9-8957-4c86-ba9b-4a7e007bde0e","Type":"ContainerDied","Data":"8b5eb1aace78f54a765c06757fa9b35e89f4f7d697fdef5ed4abee824067c672"}
Jul 24 21:05:47.821344 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.821328    2566 scope.go:117] "RemoveContainer" containerID="155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9"
Jul 24 21:05:47.821641 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:05:47.821623    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9\": container with ID starting with 155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9 not found: ID does not exist" containerID="155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9"
Jul 24 21:05:47.821701 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.821648    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9"} err="failed to get container status \"155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9\": rpc error: code = NotFound desc = could not find container \"155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9\": container with ID starting with 155305edcba2d0fcae0603da82782e8cd427b2318820c5862f2819583a48a3a9 not found: ID does not exist"
Jul 24 21:05:47.821701 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.821664    2566 scope.go:117] "RemoveContainer" containerID="81880abe4abdf91cab97cf16c26f16dd28b1059933491713f3c59763f5d77fee"
Jul 24 21:05:47.840162 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.840134    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"]
Jul 24 21:05:47.844510 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.844484    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-a-787669-6c98f988fc-mw2tf"]
Jul 24 21:05:47.861685 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.861657    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"]
Jul 24 21:05:47.868928 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:47.868860    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-787669-856dd6bf68-mmwcr"]
Jul 24 21:05:49.692495 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:49.692455    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0a9411d9-8957-4c86-ba9b-4a7e007bde0e" path="/var/lib/kubelet/pods/0a9411d9-8957-4c86-ba9b-4a7e007bde0e/volumes"
Jul 24 21:05:49.692887 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:05:49.692832    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="45017c38-2ef7-45bf-8ab1-1b93366a9f21" path="/var/lib/kubelet/pods/45017c38-2ef7-45bf-8ab1-1b93366a9f21/volumes"
Jul 24 21:06:42.233111 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233058    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"]
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233476    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="45017c38-2ef7-45bf-8ab1-1b93366a9f21" containerName="payload-processing"
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233488    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="45017c38-2ef7-45bf-8ab1-1b93366a9f21" containerName="payload-processing"
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233497    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fb831d7c-3a57-44f0-bbd2-39db30dc1775" containerName="maas-api"
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233503    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="fb831d7c-3a57-44f0-bbd2-39db30dc1775" containerName="maas-api"
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233519    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0a9411d9-8957-4c86-ba9b-4a7e007bde0e" containerName="maas-api"
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233542    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="0a9411d9-8957-4c86-ba9b-4a7e007bde0e" containerName="maas-api"
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233558    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="023116e5-5a09-41d0-b975-6bff0debd642" containerName="revoke-keys"
Jul 24 21:06:42.233566 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233566    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="023116e5-5a09-41d0-b975-6bff0debd642" containerName="revoke-keys"
Jul 24 21:06:42.233849 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233626    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="45017c38-2ef7-45bf-8ab1-1b93366a9f21" containerName="payload-processing"
Jul 24 21:06:42.233849 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233636    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="0a9411d9-8957-4c86-ba9b-4a7e007bde0e" containerName="maas-api"
Jul 24 21:06:42.233849 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233642    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="023116e5-5a09-41d0-b975-6bff0debd642" containerName="revoke-keys"
Jul 24 21:06:42.233849 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.233650    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="fb831d7c-3a57-44f0-bbd2-39db30dc1775" containerName="maas-api"
Jul 24 21:06:42.236790 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.236764    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"
Jul 24 21:06:42.239869 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.239840    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 21:06:42.248328 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.248296    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"]
Jul 24 21:06:42.369162 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.369115    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a8f977-d39b-44b3-a702-301df0db2d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw\" (UID: \"38a8f977-d39b-44b3-a702-301df0db2d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"
Jul 24 21:06:42.470344 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.470305    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a8f977-d39b-44b3-a702-301df0db2d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw\" (UID: \"38a8f977-d39b-44b3-a702-301df0db2d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"
Jul 24 21:06:42.471013 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.470991    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a8f977-d39b-44b3-a702-301df0db2d8e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw\" (UID: \"38a8f977-d39b-44b3-a702-301df0db2d8e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"
Jul 24 21:06:42.549762 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.549656    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"
Jul 24 21:06:42.695264 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:42.695238    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"]
Jul 24 21:06:42.697572 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:06:42.697495    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38a8f977_d39b_44b3_a702_301df0db2d8e.slice/crio-686c16b191ff0a70a2a51b59a8dd09841bc93cd314d3912ef8df14752eae5044 WatchSource:0}: Error finding container 686c16b191ff0a70a2a51b59a8dd09841bc93cd314d3912ef8df14752eae5044: Status 404 returned error can't find the container with id 686c16b191ff0a70a2a51b59a8dd09841bc93cd314d3912ef8df14752eae5044
Jul 24 21:06:43.100466 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:43.100428    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw" event={"ID":"38a8f977-d39b-44b3-a702-301df0db2d8e","Type":"ContainerStarted","Data":"9af6c97824e08891f02a889efd3808a9c0c1853aa4705207eda827f7dc95a938"}
Jul 24 21:06:43.100663 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:43.100512    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw" event={"ID":"38a8f977-d39b-44b3-a702-301df0db2d8e","Type":"ContainerStarted","Data":"686c16b191ff0a70a2a51b59a8dd09841bc93cd314d3912ef8df14752eae5044"}
Jul 24 21:06:43.116283 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:43.116219    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw" podStartSLOduration=1.116202624 podStartE2EDuration="1.116202624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:06:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:06:43.114597749 +0000 UTC m=+3078.040736675" watchObservedRunningTime="2026-07-24 21:06:43.116202624 +0000 UTC m=+3078.042341529"
Jul 24 21:06:44.107011 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:44.106971    2566 generic.go:356] "Generic (PLEG): container finished" podID="38a8f977-d39b-44b3-a702-301df0db2d8e" containerID="9af6c97824e08891f02a889efd3808a9c0c1853aa4705207eda827f7dc95a938" exitCode=0
Jul 24 21:06:44.107419 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:44.107052    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw" event={"ID":"38a8f977-d39b-44b3-a702-301df0db2d8e","Type":"ContainerDied","Data":"9af6c97824e08891f02a889efd3808a9c0c1853aa4705207eda827f7dc95a938"}
Jul 24 21:06:45.259814 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:45.259787    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"
Jul 24 21:06:45.397406 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:45.397311    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a8f977-d39b-44b3-a702-301df0db2d8e-maas-api-service-ca\") pod \"38a8f977-d39b-44b3-a702-301df0db2d8e\" (UID: \"38a8f977-d39b-44b3-a702-301df0db2d8e\") "
Jul 24 21:06:45.397753 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:45.397728    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a8f977-d39b-44b3-a702-301df0db2d8e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "38a8f977-d39b-44b3-a702-301df0db2d8e" (UID: "38a8f977-d39b-44b3-a702-301df0db2d8e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:06:45.498722 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:45.498679    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/38a8f977-d39b-44b3-a702-301df0db2d8e-maas-api-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:06:46.119641 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:46.119610    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"
Jul 24 21:06:46.119817 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:46.119613    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw" event={"ID":"38a8f977-d39b-44b3-a702-301df0db2d8e","Type":"ContainerDied","Data":"686c16b191ff0a70a2a51b59a8dd09841bc93cd314d3912ef8df14752eae5044"}
Jul 24 21:06:46.119817 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:46.119738    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="686c16b191ff0a70a2a51b59a8dd09841bc93cd314d3912ef8df14752eae5044"
Jul 24 21:06:52.603062 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.602956    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"]
Jul 24 21:06:52.603562 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.603207    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5" podUID="91d58022-e123-4629-b91d-46fb8f77d121" containerName="maas-api" containerID="cri-o://46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2" gracePeriod=30
Jul 24 21:06:52.868363 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.868296    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:06:52.965304 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.965273    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d58022-e123-4629-b91d-46fb8f77d121-openshift-service-ca\") pod \"91d58022-e123-4629-b91d-46fb8f77d121\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") "
Jul 24 21:06:52.965486 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.965309    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jz9fn\" (UniqueName: \"kubernetes.io/projected/91d58022-e123-4629-b91d-46fb8f77d121-kube-api-access-jz9fn\") pod \"91d58022-e123-4629-b91d-46fb8f77d121\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") "
Jul 24 21:06:52.965486 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.965332    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls\") pod \"91d58022-e123-4629-b91d-46fb8f77d121\" (UID: \"91d58022-e123-4629-b91d-46fb8f77d121\") "
Jul 24 21:06:52.965695 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.965672    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91d58022-e123-4629-b91d-46fb8f77d121-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "91d58022-e123-4629-b91d-46fb8f77d121" (UID: "91d58022-e123-4629-b91d-46fb8f77d121"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:06:52.967678 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.967641    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "91d58022-e123-4629-b91d-46fb8f77d121" (UID: "91d58022-e123-4629-b91d-46fb8f77d121"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:06:52.967829 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:52.967769    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91d58022-e123-4629-b91d-46fb8f77d121-kube-api-access-jz9fn" (OuterVolumeSpecName: "kube-api-access-jz9fn") pod "91d58022-e123-4629-b91d-46fb8f77d121" (UID: "91d58022-e123-4629-b91d-46fb8f77d121"). InnerVolumeSpecName "kube-api-access-jz9fn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:06:53.066908 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.066863    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/91d58022-e123-4629-b91d-46fb8f77d121-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:06:53.066908 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.066904    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz9fn\" (UniqueName: \"kubernetes.io/projected/91d58022-e123-4629-b91d-46fb8f77d121-kube-api-access-jz9fn\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:06:53.066908 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.066914    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/91d58022-e123-4629-b91d-46fb8f77d121-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:06:53.157346 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.157245    2566 generic.go:356] "Generic (PLEG): container finished" podID="91d58022-e123-4629-b91d-46fb8f77d121" containerID="46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2" exitCode=0
Jul 24 21:06:53.157346 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.157295    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"
Jul 24 21:06:53.157346 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.157305    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5" event={"ID":"91d58022-e123-4629-b91d-46fb8f77d121","Type":"ContainerDied","Data":"46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2"}
Jul 24 21:06:53.157756 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.157377    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5" event={"ID":"91d58022-e123-4629-b91d-46fb8f77d121","Type":"ContainerDied","Data":"024f341dcca5ec557b684247be556e05b4c450404ec19109d5507f31985fca3f"}
Jul 24 21:06:53.157756 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.157410    2566 scope.go:117] "RemoveContainer" containerID="46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2"
Jul 24 21:06:53.167210 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.167189    2566 scope.go:117] "RemoveContainer" containerID="46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2"
Jul 24 21:06:53.167584 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:06:53.167519    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2\": container with ID starting with 46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2 not found: ID does not exist" containerID="46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2"
Jul 24 21:06:53.167666 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.167594    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2"} err="failed to get container status \"46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2\": rpc error: code = NotFound desc = could not find container \"46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2\": container with ID starting with 46502e989788f417cee30845177a4f5f2809922e32ab21cd08ee3b6bde47b4b2 not found: ID does not exist"
Jul 24 21:06:53.179750 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.179725    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"]
Jul 24 21:06:53.187481 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.187457    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ipp-b-5df102-bdf89bd99-nrcc5"]
Jul 24 21:06:53.692914 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:53.692880    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="91d58022-e123-4629-b91d-46fb8f77d121" path="/var/lib/kubelet/pods/91d58022-e123-4629-b91d-46fb8f77d121/volumes"
Jul 24 21:06:57.310338 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:57.310298    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"]
Jul 24 21:06:57.312898 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:57.312868    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-5df102-47lzw"]
Jul 24 21:06:57.692579 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:06:57.692544    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="38a8f977-d39b-44b3-a702-301df0db2d8e" path="/var/lib/kubelet/pods/38a8f977-d39b-44b3-a702-301df0db2d8e/volumes"
Jul 24 21:07:04.179723 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.179683    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"]
Jul 24 21:07:04.180113 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.180089    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="38a8f977-d39b-44b3-a702-301df0db2d8e" containerName="revoke-keys"
Jul 24 21:07:04.180113 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.180101    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="38a8f977-d39b-44b3-a702-301df0db2d8e" containerName="revoke-keys"
Jul 24 21:07:04.180234 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.180120    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="91d58022-e123-4629-b91d-46fb8f77d121" containerName="maas-api"
Jul 24 21:07:04.180234 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.180128    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="91d58022-e123-4629-b91d-46fb8f77d121" containerName="maas-api"
Jul 24 21:07:04.180234 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.180191    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="91d58022-e123-4629-b91d-46fb8f77d121" containerName="maas-api"
Jul 24 21:07:04.180234 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.180199    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="38a8f977-d39b-44b3-a702-301df0db2d8e" containerName="revoke-keys"
Jul 24 21:07:04.184738 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.184718    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"
Jul 24 21:07:04.187879 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.187857    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-85rr2\""
Jul 24 21:07:04.193508 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.192360    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"]
Jul 24 21:07:04.272314 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.272276    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff8482-1316-4159-8a32-abe22763b886-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk\" (UID: \"daff8482-1316-4159-8a32-abe22763b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"
Jul 24 21:07:04.373500 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.373456    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff8482-1316-4159-8a32-abe22763b886-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk\" (UID: \"daff8482-1316-4159-8a32-abe22763b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"
Jul 24 21:07:04.374119 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.374101    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff8482-1316-4159-8a32-abe22763b886-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk\" (UID: \"daff8482-1316-4159-8a32-abe22763b886\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"
Jul 24 21:07:04.497396 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.497298    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"
Jul 24 21:07:04.641042 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:04.641009    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"]
Jul 24 21:07:04.642394 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:07:04.642369    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddaff8482_1316_4159_8a32_abe22763b886.slice/crio-c1af5e69efd5c2e0fb851e47effa6e223ca3c6c43c4e2a8ed619d8df09dfcfb1 WatchSource:0}: Error finding container c1af5e69efd5c2e0fb851e47effa6e223ca3c6c43c4e2a8ed619d8df09dfcfb1: Status 404 returned error can't find the container with id c1af5e69efd5c2e0fb851e47effa6e223ca3c6c43c4e2a8ed619d8df09dfcfb1
Jul 24 21:07:05.222956 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:05.222854    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk" event={"ID":"daff8482-1316-4159-8a32-abe22763b886","Type":"ContainerStarted","Data":"7a5da700f42a12b0e17e16070bc2ccf2b002f277c1480a214d87ca4b98c1c71c"}
Jul 24 21:07:05.222956 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:05.222913    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk" event={"ID":"daff8482-1316-4159-8a32-abe22763b886","Type":"ContainerStarted","Data":"c1af5e69efd5c2e0fb851e47effa6e223ca3c6c43c4e2a8ed619d8df09dfcfb1"}
Jul 24 21:07:05.240126 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:05.240071    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk" podStartSLOduration=1.240053592 podStartE2EDuration="1.240053592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:07:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:07:05.238890225 +0000 UTC m=+3100.165029152" watchObservedRunningTime="2026-07-24 21:07:05.240053592 +0000 UTC m=+3100.166192497"
Jul 24 21:07:06.229925 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:06.229885    2566 generic.go:356] "Generic (PLEG): container finished" podID="daff8482-1316-4159-8a32-abe22763b886" containerID="7a5da700f42a12b0e17e16070bc2ccf2b002f277c1480a214d87ca4b98c1c71c" exitCode=0
Jul 24 21:07:06.230346 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:06.229932    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk" event={"ID":"daff8482-1316-4159-8a32-abe22763b886","Type":"ContainerDied","Data":"7a5da700f42a12b0e17e16070bc2ccf2b002f277c1480a214d87ca4b98c1c71c"}
Jul 24 21:07:07.372077 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:07.372047    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"
Jul 24 21:07:07.504746 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:07.504645    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff8482-1316-4159-8a32-abe22763b886-maas-api-service-ca\") pod \"daff8482-1316-4159-8a32-abe22763b886\" (UID: \"daff8482-1316-4159-8a32-abe22763b886\") "
Jul 24 21:07:07.505028 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:07.505004    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/daff8482-1316-4159-8a32-abe22763b886-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "daff8482-1316-4159-8a32-abe22763b886" (UID: "daff8482-1316-4159-8a32-abe22763b886"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:07:07.606018 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:07.605980    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/daff8482-1316-4159-8a32-abe22763b886-maas-api-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:08.241679 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:08.241645    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"
Jul 24 21:07:08.241679 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:08.241672    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk" event={"ID":"daff8482-1316-4159-8a32-abe22763b886","Type":"ContainerDied","Data":"c1af5e69efd5c2e0fb851e47effa6e223ca3c6c43c4e2a8ed619d8df09dfcfb1"}
Jul 24 21:07:08.241898 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:08.241717    2566 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1af5e69efd5c2e0fb851e47effa6e223ca3c6c43c4e2a8ed619d8df09dfcfb1"
Jul 24 21:07:14.657568 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:14.657517    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"]
Jul 24 21:07:14.658006 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:14.657821    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd" podUID="de257412-d536-45ac-9fbe-2aa5135ff724" containerName="maas-api" containerID="cri-o://235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635" gracePeriod=30
Jul 24 21:07:14.927177 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:14.927151    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 21:07:15.081927 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.081889    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de257412-d536-45ac-9fbe-2aa5135ff724-openshift-service-ca\") pod \"de257412-d536-45ac-9fbe-2aa5135ff724\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") "
Jul 24 21:07:15.082131 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.081951    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x5c2c\" (UniqueName: \"kubernetes.io/projected/de257412-d536-45ac-9fbe-2aa5135ff724-kube-api-access-x5c2c\") pod \"de257412-d536-45ac-9fbe-2aa5135ff724\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") "
Jul 24 21:07:15.082131 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.081996    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls\") pod \"de257412-d536-45ac-9fbe-2aa5135ff724\" (UID: \"de257412-d536-45ac-9fbe-2aa5135ff724\") "
Jul 24 21:07:15.082286 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.082260    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de257412-d536-45ac-9fbe-2aa5135ff724-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "de257412-d536-45ac-9fbe-2aa5135ff724" (UID: "de257412-d536-45ac-9fbe-2aa5135ff724"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:07:15.084064 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.084038    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de257412-d536-45ac-9fbe-2aa5135ff724-kube-api-access-x5c2c" (OuterVolumeSpecName: "kube-api-access-x5c2c") pod "de257412-d536-45ac-9fbe-2aa5135ff724" (UID: "de257412-d536-45ac-9fbe-2aa5135ff724"). InnerVolumeSpecName "kube-api-access-x5c2c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:07:15.084183 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.084104    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "de257412-d536-45ac-9fbe-2aa5135ff724" (UID: "de257412-d536-45ac-9fbe-2aa5135ff724"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:07:15.183576 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.183467    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/de257412-d536-45ac-9fbe-2aa5135ff724-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:15.183576 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.183497    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x5c2c\" (UniqueName: \"kubernetes.io/projected/de257412-d536-45ac-9fbe-2aa5135ff724-kube-api-access-x5c2c\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:15.183576 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.183507    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/de257412-d536-45ac-9fbe-2aa5135ff724-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:15.278484 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.278450    2566 generic.go:356] "Generic (PLEG): container finished" podID="de257412-d536-45ac-9fbe-2aa5135ff724" containerID="235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635" exitCode=0
Jul 24 21:07:15.278707 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.278499    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"
Jul 24 21:07:15.278707 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.278559    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd" event={"ID":"de257412-d536-45ac-9fbe-2aa5135ff724","Type":"ContainerDied","Data":"235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635"}
Jul 24 21:07:15.278707 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.278612    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd" event={"ID":"de257412-d536-45ac-9fbe-2aa5135ff724","Type":"ContainerDied","Data":"06040d751c4517c7de43f4b79939500a6e25d67dd80e6da6721a402245e0755d"}
Jul 24 21:07:15.278707 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.278633    2566 scope.go:117] "RemoveContainer" containerID="235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635"
Jul 24 21:07:15.287760 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.287744    2566 scope.go:117] "RemoveContainer" containerID="235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635"
Jul 24 21:07:15.288004 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:07:15.287983    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635\": container with ID starting with 235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635 not found: ID does not exist" containerID="235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635"
Jul 24 21:07:15.288072 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.288011    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635"} err="failed to get container status \"235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635\": rpc error: code = NotFound desc = could not find container \"235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635\": container with ID starting with 235332ba92ada01ac0d5fba5fee9c3a8489918ffcdb3d4c3cdf6c939c37fc635 not found: ID does not exist"
Jul 24 21:07:15.300038 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.300016    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"]
Jul 24 21:07:15.304187 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.304168    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-3f62e6-6d44846678-xmqfd"]
Jul 24 21:07:15.693282 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:15.693252    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="de257412-d536-45ac-9fbe-2aa5135ff724" path="/var/lib/kubelet/pods/de257412-d536-45ac-9fbe-2aa5135ff724/volumes"
Jul 24 21:07:19.245628 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:19.245596    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"]
Jul 24 21:07:19.251400 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:19.251372    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-3f62e6-d9kvk"]
Jul 24 21:07:19.692642 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:19.692614    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="daff8482-1316-4159-8a32-abe22763b886" path="/var/lib/kubelet/pods/daff8482-1316-4159-8a32-abe22763b886/volumes"
Jul 24 21:07:25.568649 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:25.568617    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"]
Jul 24 21:07:25.569052 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:25.568843    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy" containerID="cri-o://b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4" gracePeriod=30
Jul 24 21:07:30.834025 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.833961    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 21:07:30.922320 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922272    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-certs\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922320 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922322    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-data\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922597 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922346    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-envoy\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922597 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922380    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70179b0f-2291-428e-ac07-04c10c47cf88-istiod-ca-cert\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922597 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922397    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-istio-token\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922597 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922451    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-credential-socket\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922597 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922475    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-socket\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922597 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922587    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sn76s\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-kube-api-access-sn76s\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922911 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922623    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70179b0f-2291-428e-ac07-04c10c47cf88-istio-podinfo\") pod \"70179b0f-2291-428e-ac07-04c10c47cf88\" (UID: \"70179b0f-2291-428e-ac07-04c10c47cf88\") "
Jul 24 21:07:30.922911 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922661    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:07:30.922911 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922788    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:07:30.922911 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922849    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:07:30.922911 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922903    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70179b0f-2291-428e-ac07-04c10c47cf88-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:07:30.923199 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922952    2566 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-certs\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:30.923199 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922969    2566 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-credential-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:30.923199 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.922983    2566 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-workload-socket\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:30.923387 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.923367    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-data" (OuterVolumeSpecName: "istio-data") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:07:30.924777 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.924748    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/70179b0f-2291-428e-ac07-04c10c47cf88-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Jul 24 21:07:30.924902 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.924808    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-kube-api-access-sn76s" (OuterVolumeSpecName: "kube-api-access-sn76s") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "kube-api-access-sn76s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:07:30.925085 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.925046    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Jul 24 21:07:30.925200 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:30.925109    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-istio-token" (OuterVolumeSpecName: "istio-token") pod "70179b0f-2291-428e-ac07-04c10c47cf88" (UID: "70179b0f-2291-428e-ac07-04c10c47cf88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:07:31.023723 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.023681    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sn76s\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-kube-api-access-sn76s\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:31.023723 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.023718    2566 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/70179b0f-2291-428e-ac07-04c10c47cf88-istio-podinfo\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:31.023723 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.023727    2566 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-data\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:31.023723 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.023736    2566 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/70179b0f-2291-428e-ac07-04c10c47cf88-istio-envoy\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:31.023990 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.023747    2566 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/70179b0f-2291-428e-ac07-04c10c47cf88-istiod-ca-cert\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:31.023990 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.023755    2566 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/70179b0f-2291-428e-ac07-04c10c47cf88-istio-token\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:31.360828 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.360793    2566 generic.go:356] "Generic (PLEG): container finished" podID="70179b0f-2291-428e-ac07-04c10c47cf88" containerID="b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4" exitCode=0
Jul 24 21:07:31.361011 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.360850    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"
Jul 24 21:07:31.361011 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.360882    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" event={"ID":"70179b0f-2291-428e-ac07-04c10c47cf88","Type":"ContainerDied","Data":"b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4"}
Jul 24 21:07:31.361011 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.360933    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96" event={"ID":"70179b0f-2291-428e-ac07-04c10c47cf88","Type":"ContainerDied","Data":"bcea5f99db22b4005f3ad1ab582d2020ba57865e0e7b36f8550fb64c3dd372e0"}
Jul 24 21:07:31.361011 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.360956    2566 scope.go:117] "RemoveContainer" containerID="b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4"
Jul 24 21:07:31.370354 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.370326    2566 scope.go:117] "RemoveContainer" containerID="303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43"
Jul 24 21:07:31.378088 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.378071    2566 scope.go:117] "RemoveContainer" containerID="b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4"
Jul 24 21:07:31.378330 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:07:31.378311    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4\": container with ID starting with b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4 not found: ID does not exist" containerID="b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4"
Jul 24 21:07:31.378383 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.378337    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4"} err="failed to get container status \"b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4\": rpc error: code = NotFound desc = could not find container \"b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4\": container with ID starting with b038ddf065c081d4fed94133a0eee43862ada137dac3e6c6e2a8d1fdc594abf4 not found: ID does not exist"
Jul 24 21:07:31.378383 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.378357    2566 scope.go:117] "RemoveContainer" containerID="303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43"
Jul 24 21:07:31.378642 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:07:31.378622    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43\": container with ID starting with 303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43 not found: ID does not exist" containerID="303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43"
Jul 24 21:07:31.378706 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.378651    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43"} err="failed to get container status \"303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43\": rpc error: code = NotFound desc = could not find container \"303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43\": container with ID starting with 303d8706ca742ffbd9e74979863d84847ab4e5729511ea8377c08e284a245e43 not found: ID does not exist"
Jul 24 21:07:31.381853 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.381832    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"]
Jul 24 21:07:31.386106 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.386087    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-3f62e6-openshift-default-5bff4bd96-lpn96"]
Jul 24 21:07:31.691905 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:31.691820    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" path="/var/lib/kubelet/pods/70179b0f-2291-428e-ac07-04c10c47cf88/volumes"
Jul 24 21:07:36.326230 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.324972    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"]
Jul 24 21:07:36.326230 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.325241    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8" podUID="d9884d50-47a7-4a93-a9d8-96a735c0c90c" containerName="maas-api" containerID="cri-o://32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0" gracePeriod=30
Jul 24 21:07:36.583068 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.583006    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 21:07:36.672943 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.672905    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bg6z\" (UniqueName: \"kubernetes.io/projected/d9884d50-47a7-4a93-a9d8-96a735c0c90c-kube-api-access-5bg6z\") pod \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") "
Jul 24 21:07:36.673107 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.672965    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls\") pod \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") "
Jul 24 21:07:36.673107 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.673085    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9884d50-47a7-4a93-a9d8-96a735c0c90c-openshift-service-ca\") pod \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\" (UID: \"d9884d50-47a7-4a93-a9d8-96a735c0c90c\") "
Jul 24 21:07:36.673414 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.673389    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9884d50-47a7-4a93-a9d8-96a735c0c90c-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "d9884d50-47a7-4a93-a9d8-96a735c0c90c" (UID: "d9884d50-47a7-4a93-a9d8-96a735c0c90c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Jul 24 21:07:36.675118 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.675088    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9884d50-47a7-4a93-a9d8-96a735c0c90c-kube-api-access-5bg6z" (OuterVolumeSpecName: "kube-api-access-5bg6z") pod "d9884d50-47a7-4a93-a9d8-96a735c0c90c" (UID: "d9884d50-47a7-4a93-a9d8-96a735c0c90c"). InnerVolumeSpecName "kube-api-access-5bg6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:07:36.675118 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.675110    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "d9884d50-47a7-4a93-a9d8-96a735c0c90c" (UID: "d9884d50-47a7-4a93-a9d8-96a735c0c90c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Jul 24 21:07:36.774667 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.774633    2566 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d9884d50-47a7-4a93-a9d8-96a735c0c90c-openshift-service-ca\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:36.774667 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.774658    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bg6z\" (UniqueName: \"kubernetes.io/projected/d9884d50-47a7-4a93-a9d8-96a735c0c90c-kube-api-access-5bg6z\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:36.774667 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:36.774669    2566 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d9884d50-47a7-4a93-a9d8-96a735c0c90c-maas-api-tls\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:37.393207 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.393176    2566 generic.go:356] "Generic (PLEG): container finished" podID="d9884d50-47a7-4a93-a9d8-96a735c0c90c" containerID="32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0" exitCode=0
Jul 24 21:07:37.393653 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.393219    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8" event={"ID":"d9884d50-47a7-4a93-a9d8-96a735c0c90c","Type":"ContainerDied","Data":"32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0"}
Jul 24 21:07:37.393653 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.393265    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8" event={"ID":"d9884d50-47a7-4a93-a9d8-96a735c0c90c","Type":"ContainerDied","Data":"ee5b80bd80966ac44fb9a7d30509f009c2d6ce203520bef1f862d92209a99a7d"}
Jul 24 21:07:37.393653 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.393286    2566 scope.go:117] "RemoveContainer" containerID="32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0"
Jul 24 21:07:37.393653 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.393220    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"
Jul 24 21:07:37.408618 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.408587    2566 scope.go:117] "RemoveContainer" containerID="32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0"
Jul 24 21:07:37.408921 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:07:37.408900    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0\": container with ID starting with 32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0 not found: ID does not exist" containerID="32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0"
Jul 24 21:07:37.408987 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.408928    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0"} err="failed to get container status \"32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0\": rpc error: code = NotFound desc = could not find container \"32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0\": container with ID starting with 32b118f0831d694f4a4b8869f72dcd707930d3ecb82a1eba9bcfeddca6ca6fe0 not found: ID does not exist"
Jul 24 21:07:37.418608 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.418585    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"]
Jul 24 21:07:37.422042 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.422019    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-449ecf-679557d4cb-hnrg8"]
Jul 24 21:07:37.693123 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:37.693031    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9884d50-47a7-4a93-a9d8-96a735c0c90c" path="/var/lib/kubelet/pods/d9884d50-47a7-4a93-a9d8-96a735c0c90c/volumes"
Jul 24 21:07:38.695590 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695549    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dxcd"]
Jul 24 21:07:38.695994 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695931    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy"
Jul 24 21:07:38.695994 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695944    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy"
Jul 24 21:07:38.695994 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695963    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9884d50-47a7-4a93-a9d8-96a735c0c90c" containerName="maas-api"
Jul 24 21:07:38.695994 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695968    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9884d50-47a7-4a93-a9d8-96a735c0c90c" containerName="maas-api"
Jul 24 21:07:38.695994 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695975    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="de257412-d536-45ac-9fbe-2aa5135ff724" containerName="maas-api"
Jul 24 21:07:38.695994 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695981    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="de257412-d536-45ac-9fbe-2aa5135ff724" containerName="maas-api"
Jul 24 21:07:38.695994 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.695994    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="daff8482-1316-4159-8a32-abe22763b886" containerName="revoke-keys"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696000    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="daff8482-1316-4159-8a32-abe22763b886" containerName="revoke-keys"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696008    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696013    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696090    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9884d50-47a7-4a93-a9d8-96a735c0c90c" containerName="maas-api"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696097    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="daff8482-1316-4159-8a32-abe22763b886" containerName="revoke-keys"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696107    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696116    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="70179b0f-2291-428e-ac07-04c10c47cf88" containerName="istio-proxy"
Jul 24 21:07:38.696221 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.696123    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="de257412-d536-45ac-9fbe-2aa5135ff724" containerName="maas-api"
Jul 24 21:07:38.700711 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.700687    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:38.706738 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.706705    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dxcd"]
Jul 24 21:07:38.792757 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.792716    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rb4nr\" (UniqueName: \"kubernetes.io/projected/90d25ca7-1f42-40a6-8624-da1012a1ef1b-kube-api-access-rb4nr\") pod \"kuadrant-operator-catalog-4dxcd\" (UID: \"90d25ca7-1f42-40a6-8624-da1012a1ef1b\") " pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:38.894268 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.894224    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rb4nr\" (UniqueName: \"kubernetes.io/projected/90d25ca7-1f42-40a6-8624-da1012a1ef1b-kube-api-access-rb4nr\") pod \"kuadrant-operator-catalog-4dxcd\" (UID: \"90d25ca7-1f42-40a6-8624-da1012a1ef1b\") " pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:38.902866 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:38.902828    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rb4nr\" (UniqueName: \"kubernetes.io/projected/90d25ca7-1f42-40a6-8624-da1012a1ef1b-kube-api-access-rb4nr\") pod \"kuadrant-operator-catalog-4dxcd\" (UID: \"90d25ca7-1f42-40a6-8624-da1012a1ef1b\") " pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:39.012479 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:39.012386    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:39.145911 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:39.145885    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dxcd"]
Jul 24 21:07:39.147820 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:07:39.147793    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90d25ca7_1f42_40a6_8624_da1012a1ef1b.slice/crio-49a5514ea7736b0ecbdf053f724e037c438b3f9f86bee72bd2608cf7071de1f9 WatchSource:0}: Error finding container 49a5514ea7736b0ecbdf053f724e037c438b3f9f86bee72bd2608cf7071de1f9: Status 404 returned error can't find the container with id 49a5514ea7736b0ecbdf053f724e037c438b3f9f86bee72bd2608cf7071de1f9
Jul 24 21:07:39.404603 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:39.404565    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd" event={"ID":"90d25ca7-1f42-40a6-8624-da1012a1ef1b","Type":"ContainerStarted","Data":"49a5514ea7736b0ecbdf053f724e037c438b3f9f86bee72bd2608cf7071de1f9"}
Jul 24 21:07:40.411162 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:40.411124    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd" event={"ID":"90d25ca7-1f42-40a6-8624-da1012a1ef1b","Type":"ContainerStarted","Data":"b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753"}
Jul 24 21:07:40.428082 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:40.428024    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd" podStartSLOduration=1.950385579 podStartE2EDuration="2.428009158s" totalImagesPullingTime="477.623579ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-07-24 21:07:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:07:40.425831049 +0000 UTC m=+3135.351969951" watchObservedRunningTime="2026-07-24 21:07:40.428009158 +0000 UTC m=+3135.354148066"
Jul 24 21:07:48.213563 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:48.213445    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-95c9487bd-96jhh_7c32077a-9e9c-4af4-a4ea-70a1adf44296/authorino/0.log"
Jul 24 21:07:49.013587 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:49.013549    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:49.013789 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:49.013610    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:49.035380 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:49.035353    2566 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:49.480271 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:49.480243    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:49.516393 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:49.516355    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dxcd"]
Jul 24 21:07:51.468602 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:51.468562    2566 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd" podUID="90d25ca7-1f42-40a6-8624-da1012a1ef1b" containerName="registry-server" containerID="cri-o://b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753" gracePeriod=2
Jul 24 21:07:51.732678 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:51.732608    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:51.817487 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:51.817447    2566 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rb4nr\" (UniqueName: \"kubernetes.io/projected/90d25ca7-1f42-40a6-8624-da1012a1ef1b-kube-api-access-rb4nr\") pod \"90d25ca7-1f42-40a6-8624-da1012a1ef1b\" (UID: \"90d25ca7-1f42-40a6-8624-da1012a1ef1b\") "
Jul 24 21:07:51.819560 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:51.819502    2566 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90d25ca7-1f42-40a6-8624-da1012a1ef1b-kube-api-access-rb4nr" (OuterVolumeSpecName: "kube-api-access-rb4nr") pod "90d25ca7-1f42-40a6-8624-da1012a1ef1b" (UID: "90d25ca7-1f42-40a6-8624-da1012a1ef1b"). InnerVolumeSpecName "kube-api-access-rb4nr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Jul 24 21:07:51.918196 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:51.918160    2566 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rb4nr\" (UniqueName: \"kubernetes.io/projected/90d25ca7-1f42-40a6-8624-da1012a1ef1b-kube-api-access-rb4nr\") on node \"ip-10-0-129-245.ec2.internal\" DevicePath \"\""
Jul 24 21:07:52.475025 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.474986    2566 generic.go:356] "Generic (PLEG): container finished" podID="90d25ca7-1f42-40a6-8624-da1012a1ef1b" containerID="b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753" exitCode=0
Jul 24 21:07:52.475450 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.475033    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd" event={"ID":"90d25ca7-1f42-40a6-8624-da1012a1ef1b","Type":"ContainerDied","Data":"b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753"}
Jul 24 21:07:52.475450 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.475049    2566 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd"
Jul 24 21:07:52.475450 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.475113    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-catalog-4dxcd" event={"ID":"90d25ca7-1f42-40a6-8624-da1012a1ef1b","Type":"ContainerDied","Data":"49a5514ea7736b0ecbdf053f724e037c438b3f9f86bee72bd2608cf7071de1f9"}
Jul 24 21:07:52.475450 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.475141    2566 scope.go:117] "RemoveContainer" containerID="b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753"
Jul 24 21:07:52.484575 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.484555    2566 scope.go:117] "RemoveContainer" containerID="b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753"
Jul 24 21:07:52.484847 ip-10-0-129-245 kubenswrapper[2566]: E0724 21:07:52.484826    2566 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753\": container with ID starting with b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753 not found: ID does not exist" containerID="b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753"
Jul 24 21:07:52.484916 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.484856    2566 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753"} err="failed to get container status \"b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753\": rpc error: code = NotFound desc = could not find container \"b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753\": container with ID starting with b5e401a2fb372eabdc203a99c2edf5c8ab723e322f18b2f9d6d6c500deb70753 not found: ID does not exist"
Jul 24 21:07:52.495860 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.495833    2566 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dxcd"]
Jul 24 21:07:52.501247 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:52.501221    2566 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-catalog-4dxcd"]
Jul 24 21:07:53.614654 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:53.614626    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5sdvn_b117a634-eb7f-49d8-84f9-0fccb697bbdf/manager/0.log"
Jul 24 21:07:53.692713 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:53.692678    2566 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="90d25ca7-1f42-40a6-8624-da1012a1ef1b" path="/var/lib/kubelet/pods/90d25ca7-1f42-40a6-8624-da1012a1ef1b/volumes"
Jul 24 21:07:53.734648 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:53.734616    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r9g9b_08120a05-023f-47c3-b1a7-04b8c880bc1e/manager/0.log"
Jul 24 21:07:53.861120 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:53.861094    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78fd9486f9-flgh8_ef6d78b9-f7e7-4909-ac84-df443ef0ace1/manager/0.log"
Jul 24 21:07:53.976341 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:53.976268    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dtkrp_54fa11de-e020-4f26-ab04-f0cadc0e4301/server/0.log"
Jul 24 21:07:54.091807 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:54.091775    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wqdt7_f702b75b-6e15-434a-92d1-c77d389d6e60/manager/0.log"
Jul 24 21:07:54.479596 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:54.479565    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c48cfcf9b-rf42p_0e1fa0e0-2caf-453a-b41a-e69f105e783b/copy-manifests/0.log"
Jul 24 21:07:54.486703 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:54.486675    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c48cfcf9b-rf42p_0e1fa0e0-2caf-453a-b41a-e69f105e783b/manager/0.log"
Jul 24 21:07:54.835826 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:54.835743    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-67d65c95c6-vvtbz_aaeb966c-2f89-4c22-a38a-6fef536e38c9/maas-api/0.log"
Jul 24 21:07:54.960063 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:54.960037    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748750-k96vp_c35b0bec-4d05-4a6b-a563-f23ecc468d4b/cleanup/0.log"
Jul 24 21:07:55.068436 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:55.068403    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748765-8tgp7_cda1a570-ea31-4768-80c8-b60383dc48c3/cleanup/0.log"
Jul 24 21:07:55.180364 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:55.180337    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29748780-9dg5c_24bbe9c0-8eb4-41f9-a872-2a9b64861fed/cleanup/0.log"
Jul 24 21:07:55.299254 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:55.299228    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-69fb4594cf-zngps_13bfd465-727d-4bac-ba4d-217febf38ddf/postgres/0.log"
Jul 24 21:07:56.598110 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:56.598070    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-95c9487bd-96jhh_7c32077a-9e9c-4af4-a4ea-70a1adf44296/authorino/0.log"
Jul 24 21:07:56.714462 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:56.714437    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-88qk8_9fddb5b2-97a3-470a-9d0a-7d562ce39cac/manager/0.log"
Jul 24 21:07:56.831631 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:56.831604    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-28x78_a485a4a5-3a3b-45f4-8949-ef5bdcaeb992/manager/0.log"
Jul 24 21:07:57.068196 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:57.068116    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6z6fb_2514a4a0-d6c1-44c3-8eb1-200208f0f627/registry-server/0.log"
Jul 24 21:07:57.190331 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:57.190296    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wj4wc_00c92151-a9bb-4bae-9040-aaf58be425f0/manager/0.log"
Jul 24 21:07:57.796845 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:57.796818    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c_2c88b65a-ebb3-4dc9-a51c-3d813671b8df/istio-proxy/0.log"
Jul 24 21:07:58.150680 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:58.150652    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fdc99b6-zff46_bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b/kube-auth-proxy/0.log"
Jul 24 21:07:58.271592 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:58.271565    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpw6c_3b7372b3-b090-433d-8e7e-f7d6299344ed/istio-proxy/1.log"
Jul 24 21:07:58.388235 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:58.388203    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hw7pp_9a1f2c26-3882-4f04-b1d6-e870df1aa010/payload-pre-processing/0.log"
Jul 24 21:07:59.082444 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:59.082414    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-npztd_7b2a8d5b-037d-48cc-a5e2-0478119db675/main/0.log"
Jul 24 21:07:59.421085 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:07:59.421051    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-74b68447df-ptcfh_d5e7a622-87da-438c-880b-a3a6fe34df2c/main/0.log"
Jul 24 21:08:08.062499 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:08.062465    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-gqxld_01d20d74-1284-4562-aaac-c289b038f9b9/global-pull-secret-syncer/0.log"
Jul 24 21:08:08.234855 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:08.234827    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-wd5fk_d07a0502-809b-4214-886b-a43cced3a8c6/konnectivity-agent/0.log"
Jul 24 21:08:08.288961 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:08.288935    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-245.ec2.internal_58be0bbdb2076ba8a2b4b2e2c5986370/haproxy/0.log"
Jul 24 21:08:13.558127 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:13.558090    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-95c9487bd-96jhh_7c32077a-9e9c-4af4-a4ea-70a1adf44296/authorino/0.log"
Jul 24 21:08:13.587340 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:13.587303    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-operator-75cc8d5cb7-88qk8_9fddb5b2-97a3-470a-9d0a-7d562ce39cac/manager/0.log"
Jul 24 21:08:13.616924 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:13.616898    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-864dd78c4c-28x78_a485a4a5-3a3b-45f4-8949-ef5bdcaeb992/manager/0.log"
Jul 24 21:08:13.683171 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:13.683140    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-catalog-6z6fb_2514a4a0-d6c1-44c3-8eb1-200208f0f627/registry-server/0.log"
Jul 24 21:08:13.850804 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:13.850775    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-86878c769f-wj4wc_00c92151-a9bb-4bae-9040-aaf58be425f0/manager/0.log"
Jul 24 21:08:16.035046 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:16.035013    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zksdl_39765225-1850-48b8-bfb8-0f0396128605/node-exporter/0.log"
Jul 24 21:08:16.057013 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:16.056987    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zksdl_39765225-1850-48b8-bfb8-0f0396128605/kube-rbac-proxy/0.log"
Jul 24 21:08:16.076373 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:16.076349    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-zksdl_39765225-1850-48b8-bfb8-0f0396128605/init-textfile/0.log"
Jul 24 21:08:18.086146 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.086108    2566 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"]
Jul 24 21:08:18.086546 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.086484    2566 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="90d25ca7-1f42-40a6-8624-da1012a1ef1b" containerName="registry-server"
Jul 24 21:08:18.086546 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.086495    2566 state_mem.go:107] "Deleted CPUSet assignment" podUID="90d25ca7-1f42-40a6-8624-da1012a1ef1b" containerName="registry-server"
Jul 24 21:08:18.086635 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.086589    2566 memory_manager.go:368] "RemoveStaleState removing state" podUID="90d25ca7-1f42-40a6-8624-da1012a1ef1b" containerName="registry-server"
Jul 24 21:08:18.089600 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.089584    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.092793 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.092771    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9s798\"/\"openshift-service-ca.crt\""
Jul 24 21:08:18.094200 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.094179    2566 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9s798\"/\"default-dockercfg-4d897\""
Jul 24 21:08:18.094320 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.094200    2566 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9s798\"/\"kube-root-ca.crt\""
Jul 24 21:08:18.098289 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.098265    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"]
Jul 24 21:08:18.141603 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.141568    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-proc\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.141603 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.141606    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8q4d\" (UniqueName: \"kubernetes.io/projected/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-kube-api-access-h8q4d\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.141824 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.141631    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-sys\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.141824 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.141655    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-lib-modules\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.141824 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.141765    2566 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-podres\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243045 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243007    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-podres\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243045 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243050    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-proc\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243265 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243069    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8q4d\" (UniqueName: \"kubernetes.io/projected/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-kube-api-access-h8q4d\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243265 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243099    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-sys\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243265 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243123    2566 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-lib-modules\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243265 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243144    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-proc\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243265 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243165    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-podres\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243265 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243230    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-sys\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.243265 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.243262    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-lib-modules\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.250605 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.250583    2566 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8q4d\" (UniqueName: \"kubernetes.io/projected/cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78-kube-api-access-h8q4d\") pod \"perf-node-gather-daemonset-xj6d5\" (UID: \"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78\") " pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.401199 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.401161    2566 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:18.532065 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.532035    2566 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"]
Jul 24 21:08:18.534616 ip-10-0-129-245 kubenswrapper[2566]: W0724 21:08:18.534588    2566 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcce4ebea_74fe_4797_9fa1_3ffb5f1c9b78.slice/crio-c87a53f756886b4f307081996c48f608573d6c6c024b651ba332d243d402b4b2 WatchSource:0}: Error finding container c87a53f756886b4f307081996c48f608573d6c6c024b651ba332d243d402b4b2: Status 404 returned error can't find the container with id c87a53f756886b4f307081996c48f608573d6c6c024b651ba332d243d402b4b2
Jul 24 21:08:18.610501 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:18.610464    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5" event={"ID":"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78","Type":"ContainerStarted","Data":"c87a53f756886b4f307081996c48f608573d6c6c024b651ba332d243d402b4b2"}
Jul 24 21:08:19.616593 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:19.616557    2566 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5" event={"ID":"cce4ebea-74fe-4797-9fa1-3ffb5f1c9b78","Type":"ContainerStarted","Data":"3533780b6beee521b4458afbc106d542b4d54b806df2b7ecd31ab28734cdc39b"}
Jul 24 21:08:19.617093 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:19.616656    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:19.633317 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:19.633263    2566 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5" podStartSLOduration=1.633249245 podStartE2EDuration="1.633249245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-07-24 21:08:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-07-24 21:08:19.632781127 +0000 UTC m=+3174.558920033" watchObservedRunningTime="2026-07-24 21:08:19.633249245 +0000 UTC m=+3174.559388151"
Jul 24 21:08:20.734873 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:20.734847    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7wtlx_df2d6053-ad84-46c0-8cf6-60497f26536f/dns/0.log"
Jul 24 21:08:20.757109 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:20.757076    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-7wtlx_df2d6053-ad84-46c0-8cf6-60497f26536f/kube-rbac-proxy/0.log"
Jul 24 21:08:20.851256 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:20.851225    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-6gllf_bda593d7-440f-45d9-a969-22ef7982c75c/dns-node-resolver/0.log"
Jul 24 21:08:21.382101 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:21.382073    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-5d66954696-sl6sp_02220d65-aa06-4bf0-8e18-5ecb31743658/registry/0.log"
Jul 24 21:08:21.449047 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:21.449017    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-qt7hz_4dcaef25-bb54-46c5-8973-8d4dd38deb93/node-ca/0.log"
Jul 24 21:08:22.343989 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:22.343956    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c_2c88b65a-ebb3-4dc9-a51c-3d813671b8df/istio-proxy/0.log"
Jul 24 21:08:22.545121 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:22.545079    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fdc99b6-zff46_bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b/kube-auth-proxy/0.log"
Jul 24 21:08:22.570828 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:22.570797    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpw6c_3b7372b3-b090-433d-8e7e-f7d6299344ed/istio-proxy/0.log"
Jul 24 21:08:22.599151 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:22.599122    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpw6c_3b7372b3-b090-433d-8e7e-f7d6299344ed/istio-proxy/1.log"
Jul 24 21:08:22.629386 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:22.629356    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hw7pp_9a1f2c26-3882-4f04-b1d6-e870df1aa010/payload-pre-processing/0.log"
Jul 24 21:08:23.491992 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:23.491955    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-2bds9_da0cd617-b89e-4636-bc16-eee42b32327b/serve-healthcheck-canary/0.log"
Jul 24 21:08:24.030309 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:24.030262    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c786m_eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0/kube-rbac-proxy/0.log"
Jul 24 21:08:24.050319 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:24.050295    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c786m_eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0/exporter/0.log"
Jul 24 21:08:24.069236 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:24.069207    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-c786m_eaa88a1d-58d6-4cf1-af2d-e81b74c4d9b0/extractor/0.log"
Jul 24 21:08:25.631993 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:25.631960    2566 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9s798/perf-node-gather-daemonset-xj6d5"
Jul 24 21:08:26.275492 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:26.275410    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-d6f4b65b8-5sdvn_b117a634-eb7f-49d8-84f9-0fccb697bbdf/manager/0.log"
Jul 24 21:08:26.474887 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:26.474852    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-r9g9b_08120a05-023f-47c3-b1a7-04b8c880bc1e/manager/0.log"
Jul 24 21:08:26.601483 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:26.601446    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-78fd9486f9-flgh8_ef6d78b9-f7e7-4909-ac84-df443ef0ace1/manager/0.log"
Jul 24 21:08:26.621172 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:26.621146    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-dtkrp_54fa11de-e020-4f26-ab04-f0cadc0e4301/server/0.log"
Jul 24 21:08:26.697410 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:26.697385    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-7998c67dc4-wqdt7_f702b75b-6e15-434a-92d1-c77d389d6e60/manager/0.log"
Jul 24 21:08:26.885660 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:26.885581    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c48cfcf9b-rf42p_0e1fa0e0-2caf-453a-b41a-e69f105e783b/manager/0.log"
Jul 24 21:08:26.904875 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:26.904849    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-7c48cfcf9b-rf42p_0e1fa0e0-2caf-453a-b41a-e69f105e783b/copy-manifests/0.log"
Jul 24 21:08:28.597337 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:28.597311    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c_2c88b65a-ebb3-4dc9-a51c-3d813671b8df/istio-proxy/0.log"
Jul 24 21:08:28.778330 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:28.778302    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fdc99b6-zff46_bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b/kube-auth-proxy/0.log"
Jul 24 21:08:28.797194 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:28.797170    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpw6c_3b7372b3-b090-433d-8e7e-f7d6299344ed/istio-proxy/0.log"
Jul 24 21:08:28.823784 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:28.823760    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpw6c_3b7372b3-b090-433d-8e7e-f7d6299344ed/istio-proxy/1.log"
Jul 24 21:08:28.845909 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:28.845885    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hw7pp_9a1f2c26-3882-4f04-b1d6-e870df1aa010/payload-pre-processing/0.log"
Jul 24 21:08:30.288461 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:30.288432    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_data-science-gateway-data-science-gateway-class-d66cd8c7c-s6w4c_2c88b65a-ebb3-4dc9-a51c-3d813671b8df/istio-proxy/0.log"
Jul 24 21:08:30.467447 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:30.467417    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-68fdc99b6-zff46_bc47ac3d-c6f6-4cdb-94ca-6cd0942d4f7b/kube-auth-proxy/0.log"
Jul 24 21:08:30.486613 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:30.486591    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpw6c_3b7372b3-b090-433d-8e7e-f7d6299344ed/istio-proxy/0.log"
Jul 24 21:08:30.511499 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:30.511477    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-78bb78ff78-kpw6c_3b7372b3-b090-433d-8e7e-f7d6299344ed/istio-proxy/1.log"
Jul 24 21:08:30.532769 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:30.532747    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-687dbb6cf6-hw7pp_9a1f2c26-3882-4f04-b1d6-e870df1aa010/payload-pre-processing/0.log"
Jul 24 21:08:34.610378 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.610344    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-9kv2f_55bcdbdd-73f7-4ebc-8a7b-7b42dcb4bb24/kube-multus/0.log"
Jul 24 21:08:34.635596 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.635569    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pz67_8efc8d87-bd58-4611-b380-47f8ddf39e81/kube-multus-additional-cni-plugins/0.log"
Jul 24 21:08:34.656252 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.656230    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pz67_8efc8d87-bd58-4611-b380-47f8ddf39e81/egress-router-binary-copy/0.log"
Jul 24 21:08:34.675709 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.675688    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pz67_8efc8d87-bd58-4611-b380-47f8ddf39e81/cni-plugins/0.log"
Jul 24 21:08:34.694417 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.694393    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pz67_8efc8d87-bd58-4611-b380-47f8ddf39e81/bond-cni-plugin/0.log"
Jul 24 21:08:34.713410 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.713389    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pz67_8efc8d87-bd58-4611-b380-47f8ddf39e81/routeoverride-cni/0.log"
Jul 24 21:08:34.731705 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.731686    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pz67_8efc8d87-bd58-4611-b380-47f8ddf39e81/whereabouts-cni-bincopy/0.log"
Jul 24 21:08:34.751274 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:34.751246    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_multus-additional-cni-plugins-2pz67_8efc8d87-bd58-4611-b380-47f8ddf39e81/whereabouts-cni/0.log"
Jul 24 21:08:35.205436 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:35.205411    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qh4xx_95607563-c0dd-46eb-a0cf-c1d1ec957617/network-metrics-daemon/0.log"
Jul 24 21:08:35.222117 ip-10-0-129-245 kubenswrapper[2566]: I0724 21:08:35.222092    2566 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-multus_network-metrics-daemon-qh4xx_95607563-c0dd-46eb-a0cf-c1d1ec957617/kube-rbac-proxy/0.log"