Sep 10 12:07:44.056688 ip-10-0-129-68 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 12:07:44.056700 ip-10-0-129-68 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 12:07:44.056709 ip-10-0-129-68 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 12:07:44.056969 ip-10-0-129-68 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 12:07:55.422482 ip-10-0-129-68 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 12:07:55.422506 ip-10-0-129-68 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c061e3efe79f43cc962deccf9dfa4000 --
Sep 10 12:09:48.695778 ip-10-0-129-68 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 12:09:49.057412 ip-10-0-129-68 kubenswrapper[2552]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:09:49.057412 ip-10-0-129-68 kubenswrapper[2552]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 12:09:49.057412 ip-10-0-129-68 kubenswrapper[2552]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:09:49.057412 ip-10-0-129-68 kubenswrapper[2552]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 12:09:49.057412 ip-10-0-129-68 kubenswrapper[2552]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 12:09:49.059825 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.059738    2552 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 12:09:49.063552 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063538    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:09:49.063552 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063552    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063556    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063560    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063563    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063567    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063570    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063573    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063576    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063579    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063581    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063584    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063587    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063590    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063592    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063595    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063598    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063601    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063603    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063606    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063608    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:09:49.063619 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063611    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063614    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063616    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063619    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063621    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063624    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063627    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063630    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063632    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063637    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063641    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063644    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063646    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063649    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063652    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063655    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063658    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063660    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063663    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:09:49.064088 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063665    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063668    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063671    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063673    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063676    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063679    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063681    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063684    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063686    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063689    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063691    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063694    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063696    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063698    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063702    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063705    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063707    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063710    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063713    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063716    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:09:49.064547 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063718    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063721    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063724    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063726    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063729    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063734    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063737    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063740    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063743    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063745    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063748    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063750    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063753    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063755    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063758    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063760    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063763    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063765    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063768    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063770    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:09:49.065032 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063773    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063776    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063780    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063782    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063785    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063788    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063791    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063795    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063798    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063801    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063803    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063806    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063808    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063811    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063813    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063816    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063819    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063821    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.063824    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:09:49.065514 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064497    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064503    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064506    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064508    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064511    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064514    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064517    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064520    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064522    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064525    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064527    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064530    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064533    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064536    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064538    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064541    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064543    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064546    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064549    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064552    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:09:49.065961 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064555    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064558    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064561    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064564    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064567    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064569    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064572    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064574    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064577    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064580    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064583    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064586    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064589    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064591    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064594    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064596    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064599    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064601    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064604    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:09:49.066509 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064606    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064609    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064612    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064614    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064617    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064619    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064622    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064625    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064627    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064630    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064633    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064636    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064638    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064642    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064645    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064647    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064650    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064653    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064656    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064658    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:09:49.066980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064661    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064664    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064666    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064670    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064672    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064675    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064678    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064680    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064683    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064685    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064688    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064690    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064693    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064695    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064698    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064700    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064705    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064708    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064712    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:09:49.067494 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064714    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064717    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064720    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064723    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064725    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064729    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064733    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064736    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064740    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064743    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064746    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064748    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064751    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064754    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064756    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064760    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064764    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064768    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064770    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:09:49.067957 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064773    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.064776    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065120    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065124    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065127    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065129    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065132    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065135    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065137    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065140    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065142    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065145    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065148    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065151    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065153    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065157    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065161    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065165    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065168    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:09:49.068546 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065170    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065173    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065176    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065178    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065181    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065183    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065186    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065188    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065191    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065193    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065196    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065199    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065202    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065205    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065208    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065210    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065213    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065216    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065218    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065221    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:09:49.069005 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065223    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065225    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065228    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065231    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065233    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065236    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065239    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065242    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065244    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065247    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065250    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065252    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065255    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065257    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065260    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065262    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065265    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065267    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065270    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065272    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:09:49.069498 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065275    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065277    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065280    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065282    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065285    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065288    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065290    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065293    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065295    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065298    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065301    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065303    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065307    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065310    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065312    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065315    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065317    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065320    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065322    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065325    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:09:49.069981 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065329    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065332    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065335    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065338    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065341    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065344    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065347    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065349    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065352    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065354    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065357    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065359    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065361    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065364    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065366    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065369    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065371    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065374    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065377    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:09:49.070496 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065379    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065382    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.065384    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065481    2552 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065516    2552 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065523    2552 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065527    2552 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065532    2552 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065535    2552 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065539    2552 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065543    2552 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065546    2552 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065549    2552 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065553    2552 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065556    2552 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065560    2552 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065563    2552 flags.go:64] FLAG: --cgroup-root=""
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065565    2552 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065569    2552 flags.go:64] FLAG: --client-ca-file=""
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065571    2552 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065574    2552 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065579    2552 flags.go:64] FLAG: --cluster-domain=""
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065582    2552 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 12:09:49.070952 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065585    2552 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065588    2552 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065591    2552 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065595    2552 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065598    2552 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065601    2552 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065604    2552 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065607    2552 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065610    2552 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065615    2552 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065618    2552 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065621    2552 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065625    2552 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065628    2552 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065631    2552 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065634    2552 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065638    2552 flags.go:64] FLAG: --enable-server="true"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065641    2552 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065645    2552 flags.go:64] FLAG: --event-burst="100"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065649    2552 flags.go:64] FLAG: --event-qps="50"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065652    2552 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065655    2552 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065657    2552 flags.go:64] FLAG: --eviction-hard=""
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065661    2552 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 12:09:49.071557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065664    2552 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065667    2552 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065670    2552 flags.go:64] FLAG: --eviction-soft=""
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065672    2552 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065675    2552 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065678    2552 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065681    2552 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065684    2552 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065687    2552 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065690    2552 flags.go:64] FLAG: --feature-gates=""
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065693    2552 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065696    2552 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065699    2552 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065702    2552 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065705    2552 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065708    2552 flags.go:64] FLAG: --help="false"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065710    2552 flags.go:64] FLAG: --hostname-override="ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065719    2552 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065723    2552 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065728    2552 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065734    2552 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065740    2552 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065744    2552 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 12:09:49.072109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065748    2552 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065751    2552 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065754    2552 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065756    2552 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065759    2552 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065762    2552 flags.go:64] FLAG: --kube-reserved=""
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065765    2552 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065768    2552 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065771    2552 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065774    2552 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065777    2552 flags.go:64] FLAG: --lock-file=""
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065779    2552 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065782    2552 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065785    2552 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065791    2552 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065793    2552 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065796    2552 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065799    2552 flags.go:64] FLAG: --logging-format="text"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065802    2552 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065805    2552 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065809    2552 flags.go:64] FLAG: --manifest-url=""
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065814    2552 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065838    2552 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065859    2552 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065872    2552 flags.go:64] FLAG: --max-pods="110"
Sep 10 12:09:49.072703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065877    2552 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065880    2552 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065883    2552 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065887    2552 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065890    2552 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065894    2552 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065897    2552 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065905    2552 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065908    2552 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065911    2552 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065914    2552 flags.go:64] FLAG: --pod-cidr=""
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065917    2552 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065923    2552 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065926    2552 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065929    2552 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065932    2552 flags.go:64] FLAG: --port="10250"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065935    2552 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065938    2552 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-085976a22e56ec9bf"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065941    2552 flags.go:64] FLAG: --qos-reserved=""
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065945    2552 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065948    2552 flags.go:64] FLAG: --register-node="true"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065951    2552 flags.go:64] FLAG: --register-with-taints=""
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065955    2552 flags.go:64] FLAG: --registry-burst="10"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065958    2552 flags.go:64] FLAG: --registry-qps="5"
Sep 10 12:09:49.073299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065960    2552 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065964    2552 flags.go:64] FLAG: --reserved-memory=""
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065968    2552 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065971    2552 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065974    2552 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065977    2552 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065981    2552 flags.go:64] FLAG: --runonce="false"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065984    2552 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065988    2552 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065991    2552 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065994    2552 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.065996    2552 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066000    2552 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066003    2552 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066006    2552 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066008    2552 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066011    2552 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066014    2552 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066017    2552 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066020    2552 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066023    2552 flags.go:64] FLAG: --system-cgroups=""
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066025    2552 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066032    2552 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066035    2552 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066037    2552 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 12:09:49.073998 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066048    2552 flags.go:64] FLAG: --tls-min-version=""
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066051    2552 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066054    2552 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066057    2552 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066060    2552 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066063    2552 flags.go:64] FLAG: --v="2"
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066066    2552 flags.go:64] FLAG: --version="false"
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066070    2552 flags.go:64] FLAG: --vmodule=""
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066076    2552 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.066079    2552 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066209    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066214    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066217    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066220    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066224    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066228    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066231    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066234    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066237    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066240    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066242    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066245    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:09:49.074992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066248    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066250    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066252    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066255    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066259    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066263    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066265    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066268    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066271    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066274    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066277    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066279    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066282    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066285    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066288    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066291    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066294    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066298    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066301    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:09:49.075685 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066303    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066306    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066309    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066312    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066315    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066321    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066324    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066327    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066329    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066332    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066334    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066337    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066339    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066342    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066345    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066347    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066350    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066352    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066355    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066359    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:09:49.076142 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066363    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066366    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066368    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066371    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066373    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066376    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066378    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066381    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066383    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066386    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066388    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066391    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066393    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066396    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066398    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066401    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066403    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066407    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066410    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066413    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:09:49.076735 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066415    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066418    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066420    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066423    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066446    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066451    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066455    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066457    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066460    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066463    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066465    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066468    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066470    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066473    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066476    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066479    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066481    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066484    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066486    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066489    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:09:49.077227 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066491    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066494    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066496    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066499    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066502    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066504    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066507    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.066510    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.067176    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.075738    2552 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.075846    2552 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.075866    2552 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.075871    2552 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075926    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075931    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:09:49.077732 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075935    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075938    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075941    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075944    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075946    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075949    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075951    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075954    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075957    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075960    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075962    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075965    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075968    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075970    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075973    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075975    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075978    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075980    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075983    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075986    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:09:49.078104 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075989    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075991    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075994    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075996    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.075999    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076002    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076005    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076009    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076013    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076017    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076020    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076023    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076026    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076029    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076032    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076034    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076037    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076040    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076042    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076045    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:09:49.078605 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076047    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076050    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076052    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076055    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076057    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076060    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076063    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076065    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076068    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076070    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076073    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076076    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076079    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076082    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076085    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076087    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076090    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076092    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076096    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:09:49.079116 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076100    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076103    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076106    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076108    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076110    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076113    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076116    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076118    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076121    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076123    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076126    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076128    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076131    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076133    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076136    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076138    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076141    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076143    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076147    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076149    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:09:49.079600 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076152    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076155    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076157    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076160    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076163    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076166    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076168    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076171    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076173    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076176    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076178    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076181    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076184    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076188    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076192    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076195    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076198    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:09:49.080070 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076201    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.076206    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076313    2552 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076317    2552 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076320    2552 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076323    2552 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076326    2552 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076329    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076331    2552 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076334    2552 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076337    2552 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076339    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076342    2552 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076344    2552 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076347    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 12:09:49.080480 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076350    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076353    2552 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076356    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076358    2552 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076361    2552 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076364    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076366    2552 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076369    2552 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076371    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076375    2552 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076379    2552 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076382    2552 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076385    2552 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076387    2552 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076390    2552 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076394    2552 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076397    2552 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076399    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076402    2552 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 12:09:49.080851 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076404    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076407    2552 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076409    2552 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076412    2552 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076414    2552 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076416    2552 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076419    2552 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076421    2552 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076442    2552 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076447    2552 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076451    2552 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076453    2552 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076456    2552 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076460    2552 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076464    2552 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076467    2552 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076470    2552 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076473    2552 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076475    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 12:09:49.081304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076478    2552 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076480    2552 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076483    2552 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076485    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076488    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076491    2552 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076493    2552 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076496    2552 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076498    2552 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076501    2552 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076503    2552 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076506    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076508    2552 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076511    2552 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076513    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076516    2552 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076518    2552 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076521    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076523    2552 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076526    2552 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 12:09:49.081773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076528    2552 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076531    2552 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076533    2552 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076535    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076538    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076540    2552 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076543    2552 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076546    2552 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076548    2552 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076551    2552 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076553    2552 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076556    2552 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076559    2552 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076561    2552 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076563    2552 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076566    2552 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076568    2552 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076571    2552 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076574    2552 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076576    2552 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 12:09:49.082246 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076578    2552 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076581    2552 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076583    2552 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076586    2552 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076588    2552 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076591    2552 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076593    2552 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:49.076596    2552 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.076601    2552 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.076698    2552 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.079985    2552 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.080855    2552 server.go:1019] "Starting client certificate rotation"
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.080949    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 12:09:49.082741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.081630    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 12:09:49.083887 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.083874    2552 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 12:09:49.086473 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.086452    2552 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 12:09:49.095313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.095297    2552 log.go:25] "Validated CRI v1 runtime API"
Sep 10 12:09:49.098292 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.098276    2552 log.go:25] "Validated CRI v1 image API"
Sep 10 12:09:49.100333 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.100317    2552 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 12:09:49.103085 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.103067    2552 fs.go:135] Filesystem UUIDs: map[119e6cbb-0803-4bb2-b7ca-0061aadffb00:/dev/nvme0n1p4 12f77e97-b2f6-4ea6-b178-1cc034900f84:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 10 12:09:49.103166 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.103088    2552 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 12:09:49.108761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.108638    2552 manager.go:216] Machine: {Timestamp:2026-09-10 12:09:49.107250134 +0000 UTC m=+0.303765825 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3101684 MemoryCapacity:32812163072 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f34d797569b4c672d1c4a12836f81 SystemUUID:ec2f34d7-9756-9b4c-672d-1c4a12836f81 BootID:c061e3ef-e79f-43cc-962d-eccf9dfa4000 Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406081536 Type:vfs Inodes:4005391 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:39:e1:8a:5e:7d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:39:e1:8a:5e:7d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:c2:d6:55:8a:3d:11 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812163072 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 12:09:49.108761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.108755    2552 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 12:09:49.108879 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.108852    2552 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 12:09:49.111307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.111284    2552 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 12:09:49.111473 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.111319    2552 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-68.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 12:09:49.112660 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.112648    2552 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 12:09:49.112706 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.112662    2552 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 12:09:49.112706 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.112670    2552 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 12:09:49.112706 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.112687    2552 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 12:09:49.113704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.113686    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 12:09:49.114209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.114197    2552 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 12:09:49.118247 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.118235    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 12:09:49.118357 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.118348    2552 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 12:09:49.118392 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.118377    2552 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 12:09:49.118392 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.118385    2552 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 12:09:49.118463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.118449    2552 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 12:09:49.118463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.118462    2552 kubelet.go:398] "Adding apiserver pod source"
Sep 10 12:09:49.118517 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.118473    2552 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 12:09:49.120098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.120025    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 12:09:49.120197 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.120105    2552 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 12:09:49.122518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.122495    2552 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 12:09:49.125115 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125101    2552 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 12:09:49.125179 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125132    2552 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 12:09:49.125284 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125273    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 12:09:49.125317 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125294    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 12:09:49.125317 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125306    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 12:09:49.125317 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125314    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 12:09:49.125419 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125322    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 12:09:49.125419 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125328    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 12:09:49.125856 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125847    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 12:09:49.125885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125857    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 12:09:49.125885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125864    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 12:09:49.125885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125869    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 12:09:49.125885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125877    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 12:09:49.125885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.125885    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 12:09:49.126467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.126457    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 12:09:49.126504 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.126468    2552 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 12:09:49.130125 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.130109    2552 server.go:1268] "Started kubelet"
Sep 10 12:09:49.130232 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.130179    2552 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 12:09:49.130949 ip-10-0-129-68 systemd[1]: Started Kubernetes Kubelet.
Sep 10 12:09:49.131273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.130926    2552 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 12:09:49.131273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.130986    2552 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 12:09:49.132180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.132164    2552 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 12:09:49.132463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.132451    2552 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 12:09:49.133722 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.133693    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 12:09:49.133805 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.133729    2552 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-68.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 12:09:49.133805 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.133782    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 12:09:49.135116 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.135084    2552 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 12:09:49.136318 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.135404    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646b491905  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.130078469 +0000 UTC m=+0.326594161,LastTimestamp:2026-09-10 12:09:49.130078469 +0000 UTC m=+0.326594161,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.136854 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.136837    2552 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 12:09:49.137397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.137385    2552 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 12:09:49.137601 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.137562    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.138104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138091    2552 factory.go:55] Registering systemd factory
Sep 10 12:09:49.138157 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138111    2552 factory.go:223] Registration of the systemd container factory successfully
Sep 10 12:09:49.138206 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138183    2552 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 12:09:49.138206 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138198    2552 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 12:09:49.138206 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138183    2552 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 12:09:49.138345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138307    2552 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 12:09:49.138345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138314    2552 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 12:09:49.138486 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138474    2552 factory.go:153] Registering CRI-O factory
Sep 10 12:09:49.138524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138488    2552 factory.go:223] Registration of the crio container factory successfully
Sep 10 12:09:49.138561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138527    2552 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 12:09:49.138561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138539    2552 factory.go:103] Registering Raw factory
Sep 10 12:09:49.138561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138551    2552 manager.go:1196] Started watching for new ooms in manager
Sep 10 12:09:49.138875 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.138864    2552 manager.go:349] Starting recovery of all containers
Sep 10 12:09:49.150494 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.150473    2552 manager.go:354] Recovery completed
Sep 10 12:09:49.151362 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.151328    2552 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-68.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 12:09:49.151818 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.151778    2552 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 12:09:49.156635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.156623    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:49.159307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.159289    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:49.159397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.159320    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:49.159397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.159331    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:49.159940 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.159924    2552 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 12:09:49.159940 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.159938    2552 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 12:09:49.160037 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.159954    2552 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 12:09:49.160037 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.159955    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:49.160667 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.160654    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:49.160726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.160672    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:49.160726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.160682    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:49.162003 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.161939    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646d070f85  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-68.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.159305093 +0000 UTC m=+0.355820789,LastTimestamp:2026-09-10 12:09:49.159305093 +0000 UTC m=+0.355820789,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.162116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.162103    2552 policy_none.go:49] "None policy: Start"
Sep 10 12:09:49.162153 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.162124    2552 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 12:09:49.162153 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.162140    2552 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 12:09:49.164131 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.164120    2552 policy_none.go:47] "Start"
Sep 10 12:09:49.170953 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.170875    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646d075f04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-68.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.159325444 +0000 UTC m=+0.355841135,LastTimestamp:2026-09-10 12:09:49.159325444 +0000 UTC m=+0.355841135,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.176304 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.176245    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646d07a3ef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-68.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.159343087 +0000 UTC m=+0.355858903,LastTimestamp:2026-09-10 12:09:49.159343087 +0000 UTC m=+0.355858903,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.181227 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.181147    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-68.ec2.internal.18d3f4646d070f85\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646d070f85  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-68.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.159305093 +0000 UTC m=+0.355820789,LastTimestamp:2026-09-10 12:09:49.160665057 +0000 UTC m=+0.357180748,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.186447 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.186376    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-68.ec2.internal.18d3f4646d075f04\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646d075f04  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-68.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.159325444 +0000 UTC m=+0.355841135,LastTimestamp:2026-09-10 12:09:49.160676513 +0000 UTC m=+0.357192207,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.196227    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-98gm6"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.197529    2552 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-68.ec2.internal.18d3f4646d07a3ef\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646d07a3ef  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-68.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.159343087 +0000 UTC m=+0.355858903,LastTimestamp:2026-09-10 12:09:49.16068562 +0000 UTC m=+0.357201311,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.198961    2552 manager.go:337] "Starting Device Plugin manager"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.198989    2552 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.198997    2552 server.go:87] "Starting device plugin registration server"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.199199    2552 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.199208    2552 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.199323    2552 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.199492    2552 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.199500    2552 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.199838    2552 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.199887    2552 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.209209    2552 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-68.ec2.internal.18d3f4646f82f18c  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-68.ec2.internal,UID:ip-10-0-129-68.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeAllocatableEnforced,Message:Updated Node Allocatable limit across pods,Source:EventSource{Component:kubelet,Host:ip-10-0-129-68.ec2.internal,},FirstTimestamp:2026-09-10 12:09:49.200978316 +0000 UTC m=+0.397493994,LastTimestamp:2026-09-10 12:09:49.200978316 +0000 UTC m=+0.397493994,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-68.ec2.internal,}"
Sep 10 12:09:49.214349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.211108    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-98gm6"
Sep 10 12:09:49.234628 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.234601    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 12:09:49.235890 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.235876    2552 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 12:09:49.235953 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.235918    2552 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 12:09:49.235953 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.235935    2552 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 12:09:49.236042 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.235966    2552 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 12:09:49.247223 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.247208    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:09:49.299941 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.299918    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:49.300751 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.300736    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:49.300852 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.300767    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:49.300852 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.300782    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:49.300852 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.300809    2552 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.312801 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.312748    2552 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.312801 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.312767    2552 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-68.ec2.internal\": node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.336039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.336018    2552 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal"]
Sep 10 12:09:49.336117 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.336098    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:49.337537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.337522    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:49.337595 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.337550    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:49.337595 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.337560    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:49.339731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.339712    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e98781e10c3db8f0fd9a7eebd64a1c91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal\" (UID: \"e98781e10c3db8f0fd9a7eebd64a1c91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.339798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.339744    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98781e10c3db8f0fd9a7eebd64a1c91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal\" (UID: \"e98781e10c3db8f0fd9a7eebd64a1c91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.339798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.339788    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:49.339898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.339881    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.339935 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.339904    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:49.340492 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.340478    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:49.340558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.340489    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:49.340558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.340503    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:49.340558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.340508    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:49.340558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.340518    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:49.340558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.340522    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:49.342654 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.342638    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.342730 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.342667    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:49.343314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.343297    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:49.343468 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.343324    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:49.343468 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.343344    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:49.344628 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.344612    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.383045 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.383026    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-68.ec2.internal\" not found" node="ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.387230 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.387216    2552 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-68.ec2.internal\" not found" node="ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.440078 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.440062    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e98781e10c3db8f0fd9a7eebd64a1c91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal\" (UID: \"e98781e10c3db8f0fd9a7eebd64a1c91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.440159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.440084    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98781e10c3db8f0fd9a7eebd64a1c91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal\" (UID: \"e98781e10c3db8f0fd9a7eebd64a1c91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.440159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.440102    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d08dde1ef99723fcb5d6e25dbc85b3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-68.ec2.internal\" (UID: \"72d08dde1ef99723fcb5d6e25dbc85b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.440159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.440133    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/e98781e10c3db8f0fd9a7eebd64a1c91-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal\" (UID: \"e98781e10c3db8f0fd9a7eebd64a1c91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.440159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.440153    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e98781e10c3db8f0fd9a7eebd64a1c91-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal\" (UID: \"e98781e10c3db8f0fd9a7eebd64a1c91\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.445644 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.445626    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.540458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.540421    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d08dde1ef99723fcb5d6e25dbc85b3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-68.ec2.internal\" (UID: \"72d08dde1ef99723fcb5d6e25dbc85b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.540524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.540467    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/72d08dde1ef99723fcb5d6e25dbc85b3-config\") pod \"kube-apiserver-proxy-ip-10-0-129-68.ec2.internal\" (UID: \"72d08dde1ef99723fcb5d6e25dbc85b3\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.546559 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.546540    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.647392 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.647350    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.686290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.686275    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.691210 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:49.691128    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:49.748374 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.748354    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.848896 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.848867    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:49.949419 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:49.949373    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:50.049863 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:50.049835    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:50.080996 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.080978    2552 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 12:09:50.081374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.081131    2552 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 12:09:50.150645 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:50.150620    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:50.214599 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.214526    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 12:04:49 +0000 UTC" deadline="2028-06-16 18:00:35.293411836 +0000 UTC"
Sep 10 12:09:50.214599 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.214569    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15485h50m45.078845739s"
Sep 10 12:09:50.250847 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:50.250821    2552 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-68.ec2.internal\" not found"
Sep 10 12:09:50.298773 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:50.298640    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72d08dde1ef99723fcb5d6e25dbc85b3.slice/crio-ca89ed4ea45f9960ca331052b2860d5fc098e878fdef60a841a49766f037a9d0 WatchSource:0}: Error finding container ca89ed4ea45f9960ca331052b2860d5fc098e878fdef60a841a49766f037a9d0: Status 404 returned error can't find the container with id ca89ed4ea45f9960ca331052b2860d5fc098e878fdef60a841a49766f037a9d0
Sep 10 12:09:50.298967 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:50.298951    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode98781e10c3db8f0fd9a7eebd64a1c91.slice/crio-04827a342b15f7662ce48e4e46d849cd7c36183e2a438554d82360442184ed15 WatchSource:0}: Error finding container 04827a342b15f7662ce48e4e46d849cd7c36183e2a438554d82360442184ed15: Status 404 returned error can't find the container with id 04827a342b15f7662ce48e4e46d849cd7c36183e2a438554d82360442184ed15
Sep 10 12:09:50.302134 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.302116    2552 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 12:09:50.302982 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.302954    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 12:09:50.303033 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.302991    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 12:09:50.303033 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.303006    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeHasSufficientPID"
Sep 10 12:09:50.304868 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.304849    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:09:50.311733 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.311717    2552 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:09:50.338559 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.338541    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:50.354086 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.354073    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 12:09:50.355609 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.355597    2552 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal"
Sep 10 12:09:50.370417 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.370402    2552 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 12:09:50.443180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.443158    2552 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:09:50.516766 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:50.516722    2552 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:09:51.120315 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.120238    2552 apiserver.go:52] "Watching apiserver"
Sep 10 12:09:51.129870 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.129844    2552 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 12:09:51.130315 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.130284    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/node-ca-f8nkn","openshift-network-diagnostics/network-check-target-hdnv2","openshift-network-operator/iptables-alerter-rrmhh","openshift-cluster-node-tuning-operator/tuned-f2b6g","openshift-dns/node-resolver-rfpvc","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal","openshift-multus/multus-additional-cni-plugins-pxwnc","openshift-multus/multus-mgklt","openshift-multus/network-metrics-daemon-cxj2p","openshift-ovn-kubernetes/ovnkube-node-mhb45","kube-system/global-pull-secret-syncer-w7g5k","kube-system/konnectivity-agent-gdnqz","kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"]
Sep 10 12:09:51.133327 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.133300    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.135598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.135572    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.137445 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.137399    2552 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 12:09:51.138144 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.138126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.138495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.138460    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 12:09:51.138613 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.138567    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 12:09:51.138775 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.138754    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-r9rk5\""
Sep 10 12:09:51.140053 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.140033    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 12:09:51.140141 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.140040    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 12:09:51.140270 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.140252    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 12:09:51.140808 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.140584    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.143734 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.142871    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.143734 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.143090    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 12:09:51.143734 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.143392    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-s7b5q\""
Sep 10 12:09:51.143734 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.143434    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 12:09:51.145676 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.144557    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:09:51.145676 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.145256    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.146209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.146184    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:09:51.146420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.146344    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 12:09:51.146605 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.146589    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-brggr\""
Sep 10 12:09:51.147316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.147299    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-8fdbx\""
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.147786    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.147963    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-6frm2\""
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148014    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148192    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e96c83bb-d317-452c-8326-0e996ab29ebf-agent-certs\") pod \"konnectivity-agent-gdnqz\" (UID: \"e96c83bb-d317-452c-8326-0e996ab29ebf\") " pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148220    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e96c83bb-d317-452c-8326-0e996ab29ebf-konnectivity-ca\") pod \"konnectivity-agent-gdnqz\" (UID: \"e96c83bb-d317-452c-8326-0e996ab29ebf\") " pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148249    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-modprobe-d\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148274    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysctl-conf\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148299    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-lib-modules\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-cnibin\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148357    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148381    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148402    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148408    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-systemd\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148434    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-tuned\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148462    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jzglt\" (UniqueName: \"kubernetes.io/projected/17141246-0ee1-4172-8147-f2ace482de53-kube-api-access-jzglt\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148487    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-var-lib-kubelet\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148533    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-sys\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148557    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psx7m\" (UniqueName: \"kubernetes.io/projected/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-kube-api-access-psx7m\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.149549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148596    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-kubernetes\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148640    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysctl-d\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148672    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-run\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148702    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-tmp\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148727    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06141519-5f08-4c52-80bb-1564b403251b-iptables-alerter-script\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148757    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-system-cni-dir\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148795    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-cni-binary-copy\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148822    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148848    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysconfig\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148893    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06141519-5f08-4c52-80bb-1564b403251b-host-slash\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148301    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148929    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-trsw8\" (UniqueName: \"kubernetes.io/projected/06141519-5f08-4c52-80bb-1564b403251b-kube-api-access-trsw8\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.148974    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-host\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.149115    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:51.150495 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.149199    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:09:51.151873 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.151612    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.154622 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.152601    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 12:09:51.154622 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.152632    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-mz2xh\""
Sep 10 12:09:51.154622 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.152632    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 12:09:51.154622 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.152634    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 12:09:51.157318 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.157297    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:51.157417 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.157363    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:09:51.162280 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.160286    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 12:09:51.162280 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.160422    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.162280 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.160556    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 12:09:51.162280 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.160935    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-pmxqr\""
Sep 10 12:09:51.162280 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.161099    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 12:09:51.167813 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.167789    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.167903 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.167880    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:09:51.168773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.168751    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 12:09:51.170247 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.170221    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.171425 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.171405    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-29jdx\""
Sep 10 12:09:51.181007 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.180985    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 12:09:51.181007 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.181007    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 12:09:51.181242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.180988    2552 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 12:09:51.181242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.181111    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 12:09:51.181415 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.181398    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 12:09:51.181586 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.181570    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 12:09:51.182173 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.182156    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 12:09:51.182445 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.182156    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-crvvv\""
Sep 10 12:09:51.228868 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.228847    2552 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-b6872"
Sep 10 12:09:51.240051 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.240030    2552 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 12:09:51.241731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.241682    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal" event={"ID":"e98781e10c3db8f0fd9a7eebd64a1c91","Type":"ContainerStarted","Data":"04827a342b15f7662ce48e4e46d849cd7c36183e2a438554d82360442184ed15"}
Sep 10 12:09:51.243225 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.243198    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal" event={"ID":"72d08dde1ef99723fcb5d6e25dbc85b3","Type":"ContainerStarted","Data":"ca89ed4ea45f9960ca331052b2860d5fc098e878fdef60a841a49766f037a9d0"}
Sep 10 12:09:51.244259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.244239    2552 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-b6872"
Sep 10 12:09:51.249932 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.249909    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-ovn\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.250035 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.249947    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.250035 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.249975    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovnkube-config\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.250035 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250009    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e96c83bb-d317-452c-8326-0e996ab29ebf-agent-certs\") pod \"konnectivity-agent-gdnqz\" (UID: \"e96c83bb-d317-452c-8326-0e996ab29ebf\") " pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.250195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250045    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-modprobe-d\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.250195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250071    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-lib-modules\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.250195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250097    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:51.250195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250132    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c982e9cd-3e88-4e3e-9d59-4bbca5053412-dbus\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.250195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250155    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-kubelet\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.250195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250179    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-systemd\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250203    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-var-lib-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250240    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250283    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250307    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-systemd\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250329    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-cni-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250350    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-conf-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250398    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-daemon-config\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250502    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vq4pq\" (UniqueName: \"kubernetes.io/projected/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-kube-api-access-vq4pq\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.250525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250524    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-node-log\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jzglt\" (UniqueName: \"kubernetes.io/projected/17141246-0ee1-4172-8147-f2ace482de53-kube-api-access-jzglt\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250557    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-modprobe-d\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250570    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-hosts-file\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250598    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-registration-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250618    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-cni-binary-copy\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250657    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-socket-dir-parent\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250682    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-var-lib-kubelet\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250705    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-tuning-conf-dir\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250729    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-sys-fs\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250772    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-hostroot\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250788    2552 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250798    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c982e9cd-3e88-4e3e-9d59-4bbca5053412-kubelet-config\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250806    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-lib-modules\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.250884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250823    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-slash\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250938    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-systemd\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250965    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-cni-bin\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.250993    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-env-overrides\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251013    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovnkube-script-lib\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251041    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-sys\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251093    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-sys\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251120    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-var-lib-kubelet\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kdn49\" (UniqueName: \"kubernetes.io/projected/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-kube-api-access-kdn49\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251166    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-cni-netd\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251192    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-kubernetes\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251215    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysctl-d\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251255    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251293    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7qn6p\" (UniqueName: \"kubernetes.io/projected/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-kube-api-access-7qn6p\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251322    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysctl-d\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251344    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251349    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/461c173a-3009-4c70-a4c1-ba0ca3e8b945-host\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.251583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251386    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-tmp\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251380    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-kubernetes\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251421    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251448    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysconfig\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251481    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-systemd-units\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251523    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06141519-5f08-4c52-80bb-1564b403251b-host-slash\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251534    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysconfig\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251570    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/06141519-5f08-4c52-80bb-1564b403251b-host-slash\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251579    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7k9\" (UniqueName: \"kubernetes.io/projected/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-kube-api-access-tb7k9\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251627    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8f4gw\" (UniqueName: \"kubernetes.io/projected/461c173a-3009-4c70-a4c1-ba0ca3e8b945-kube-api-access-8f4gw\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251663    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-cni-multus\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251698    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e96c83bb-d317-452c-8326-0e996ab29ebf-konnectivity-ca\") pod \"konnectivity-agent-gdnqz\" (UID: \"e96c83bb-d317-452c-8326-0e996ab29ebf\") " pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251730    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysctl-conf\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251757    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/461c173a-3009-4c70-a4c1-ba0ca3e8b945-serviceca\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251872    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-cni-bin\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251900    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-multus-certs\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251956    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-run-netns\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.252265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.251958    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252017    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovn-node-metrics-cert\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252045    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-cnibin\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252069    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252156    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-log-socket\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252181    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-tuned\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252208    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-netns\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252255    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/e96c83bb-d317-452c-8326-0e996ab29ebf-konnectivity-ca\") pod \"konnectivity-agent-gdnqz\" (UID: \"e96c83bb-d317-452c-8326-0e996ab29ebf\") " pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252260    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfdmh\" (UniqueName: \"kubernetes.io/projected/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-kube-api-access-kfdmh\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252286    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-tmp-dir\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-etc-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252340    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252430    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psx7m\" (UniqueName: \"kubernetes.io/projected/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-kube-api-access-psx7m\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252454    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-kubelet\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252471    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.253207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252487    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-device-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252453    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-cnibin\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252519    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-socket-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252556    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-system-cni-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252575    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-run\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252595    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06141519-5f08-4c52-80bb-1564b403251b-iptables-alerter-script\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252611    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-sysctl-conf\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252628    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-system-cni-dir\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252791    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-cni-binary-copy\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252828    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252858    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-trsw8\" (UniqueName: \"kubernetes.io/projected/06141519-5f08-4c52-80bb-1564b403251b-kube-api-access-trsw8\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252891    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-host\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252904    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-run\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252920    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-cnibin\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252933    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/17141246-0ee1-4172-8147-f2ace482de53-system-cni-dir\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.252946    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-k8s-cni-cncf-io\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.253028    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-etc-kubernetes\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.254074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.253408    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/06141519-5f08-4c52-80bb-1564b403251b-iptables-alerter-script\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.254937 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.253638    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-host\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.254937 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.253739    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/17141246-0ee1-4172-8147-f2ace482de53-cni-binary-copy\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.255446 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.255423    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-etc-tuned\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.255920 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.255903    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-tmp\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.256003 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.255945    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/e96c83bb-d317-452c-8326-0e996ab29ebf-agent-certs\") pod \"konnectivity-agent-gdnqz\" (UID: \"e96c83bb-d317-452c-8326-0e996ab29ebf\") " pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.268334 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.268306    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-trsw8\" (UniqueName: \"kubernetes.io/projected/06141519-5f08-4c52-80bb-1564b403251b-kube-api-access-trsw8\") pod \"iptables-alerter-rrmhh\" (UID: \"06141519-5f08-4c52-80bb-1564b403251b\") " pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.271898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.271871    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jzglt\" (UniqueName: \"kubernetes.io/projected/17141246-0ee1-4172-8147-f2ace482de53-kube-api-access-jzglt\") pod \"multus-additional-cni-plugins-pxwnc\" (UID: \"17141246-0ee1-4172-8147-f2ace482de53\") " pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.288406 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.288387    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psx7m\" (UniqueName: \"kubernetes.io/projected/f07e4783-bae9-46c7-88c8-4e8ad5c0b37e-kube-api-access-psx7m\") pod \"tuned-f2b6g\" (UID: \"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e\") " pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.334844 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.334818    2552 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353548    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-conf-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353589    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-daemon-config\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353614    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vq4pq\" (UniqueName: \"kubernetes.io/projected/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-kube-api-access-vq4pq\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353638    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-node-log\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353698    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-node-log\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353720    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-conf-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353733    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-hosts-file\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353762    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-registration-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353789    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-cni-binary-copy\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353814    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-socket-dir-parent\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353840    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-sys-fs\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353863    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-hostroot\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353870    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-registration-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353900    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c982e9cd-3e88-4e3e-9d59-4bbca5053412-kubelet-config\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353905    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-hosts-file\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353948    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-sys-fs\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353970    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-hostroot\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.354026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353972    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c982e9cd-3e88-4e3e-9d59-4bbca5053412-kubelet-config\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353992    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-slash\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.353925    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-slash\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354021    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-socket-dir-parent\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354032    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-cni-bin\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354071    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-cni-bin\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354063    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-env-overrides\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354234    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-daemon-config\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354244    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovnkube-script-lib\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354273    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kdn49\" (UniqueName: \"kubernetes.io/projected/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-kube-api-access-kdn49\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354297    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-cni-netd\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354321    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354345    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7qn6p\" (UniqueName: \"kubernetes.io/projected/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-kube-api-access-7qn6p\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354360    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-cni-netd\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354392    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/461c173a-3009-4c70-a4c1-ba0ca3e8b945-host\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354421    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-systemd-units\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354499    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7k9\" (UniqueName: \"kubernetes.io/projected/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-kube-api-access-tb7k9\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:51.355471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354544    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8f4gw\" (UniqueName: \"kubernetes.io/projected/461c173a-3009-4c70-a4c1-ba0ca3e8b945-kube-api-access-8f4gw\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354538    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-env-overrides\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354572    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-cni-multus\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354607    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/461c173a-3009-4c70-a4c1-ba0ca3e8b945-serviceca\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354622    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-cni-multus\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354626    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-cni-bin\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354316    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-cni-binary-copy\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354650    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-multus-certs\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354629    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354692    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-cni-bin\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354693    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-multus-certs\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354718    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/461c173a-3009-4c70-a4c1-ba0ca3e8b945-host\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354719    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-systemd-units\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354740    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-run-netns\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354766    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovn-node-metrics-cert\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354787    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovnkube-script-lib\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354800    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354818    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:51.356420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354853    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-log-socket\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-netns\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354886    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfdmh\" (UniqueName: \"kubernetes.io/projected/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-kube-api-access-kfdmh\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354931    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-run-netns\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.354987    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-log-socket\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355044    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/461c173a-3009-4c70-a4c1-ba0ca3e8b945-serviceca\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355055    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-etc-selinux\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355059    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-netns\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355088    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-tmp-dir\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355105    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-etc-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355122    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355140    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-kubelet\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355149    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-etc-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355156    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355173    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-device-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355184    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355212    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-var-lib-kubelet\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.357497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355246    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-kubelet-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355274    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-device-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355188    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-socket-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355284    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-socket-dir\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355295    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-system-cni-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355315    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355331    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-cnibin\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355348    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-k8s-cni-cncf-io\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355364    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-etc-kubernetes\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355380    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-ovn\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355380    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-tmp-dir\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355394    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355422    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-run-ovn-kubernetes\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355430    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovnkube-config\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355445    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-system-cni-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355451    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-cnibin\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355472    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-host-run-k8s-cni-cncf-io\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355460    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:51.358332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355500    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c982e9cd-3e88-4e3e-9d59-4bbca5053412-dbus\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.355529    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355549    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-kubelet\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.355592    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355618    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-etc-kubernetes\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355676    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c982e9cd-3e88-4e3e-9d59-4bbca5053412-dbus\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355594    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-ovn\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355728    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-host-kubelet\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.355597    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret podName:c982e9cd-3e88-4e3e-9d59-4bbca5053412 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:51.855573151 +0000 UTC m=+3.052088866 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret") pod "global-pull-secret-syncer-w7g5k" (UID: "c982e9cd-3e88-4e3e-9d59-4bbca5053412") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.355910    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-systemd\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.355972    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs podName:2948f2cc-e57c-48e7-9f5b-c4efd28fd928 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:51.855959244 +0000 UTC m=+3.052474923 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs") pod "network-metrics-daemon-cxj2p" (UID: "2948f2cc-e57c-48e7-9f5b-c4efd28fd928") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.356000    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-run-systemd\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.356068    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-var-lib-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.356145    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-var-lib-openvswitch\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.356157    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-cni-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.356285    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-multus-cni-dir\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.356415    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovnkube-config\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.359301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.357985    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-ovn-node-metrics-cert\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.381956 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.381901    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vq4pq\" (UniqueName: \"kubernetes.io/projected/e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f-kube-api-access-vq4pq\") pod \"multus-mgklt\" (UID: \"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f\") " pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.399836 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.399810    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8f4gw\" (UniqueName: \"kubernetes.io/projected/461c173a-3009-4c70-a4c1-ba0ca3e8b945-kube-api-access-8f4gw\") pod \"node-ca-f8nkn\" (UID: \"461c173a-3009-4c70-a4c1-ba0ca3e8b945\") " pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.405560 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.405535    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7k9\" (UniqueName: \"kubernetes.io/projected/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-kube-api-access-tb7k9\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:51.405652 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.405576    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:09:51.405652 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.405595    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:09:51.405652 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.405610    2552 projected.go:196] Error preparing data for projected volume kube-api-access-2jgnh for pod openshift-network-diagnostics/network-check-target-hdnv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:51.405857 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.405683    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh podName:ee0458f0-6820-424d-b378-1424b0bb7e24 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:51.90566494 +0000 UTC m=+3.102180637 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-2jgnh" (UniqueName: "kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh") pod "network-check-target-hdnv2" (UID: "ee0458f0-6820-424d-b378-1424b0bb7e24") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:51.406870 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.406846    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7qn6p\" (UniqueName: \"kubernetes.io/projected/bbfe1596-ceaa-4f73-aa21-59bb62bf1527-kube-api-access-7qn6p\") pod \"node-resolver-rfpvc\" (UID: \"bbfe1596-ceaa-4f73-aa21-59bb62bf1527\") " pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.409682 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.409654    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kdn49\" (UniqueName: \"kubernetes.io/projected/e071ba5f-61f8-4bde-9a1a-f41d19b319c8-kube-api-access-kdn49\") pod \"aws-ebs-csi-driver-node-vxv5d\" (UID: \"e071ba5f-61f8-4bde-9a1a-f41d19b319c8\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.410766 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.410742    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfdmh\" (UniqueName: \"kubernetes.io/projected/f1b5a6d7-3ac8-400a-ac94-c332aed0ee59-kube-api-access-kfdmh\") pod \"ovnkube-node-mhb45\" (UID: \"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59\") " pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.448168 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.448132    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-pxwnc"
Sep 10 12:09:51.457452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.457431    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:09:51.478715 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.478689    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-rrmhh"
Sep 10 12:09:51.483836 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.483806    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-f2b6g"
Sep 10 12:09:51.487388 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.487367    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode96c83bb_d317_452c_8326_0e996ab29ebf.slice/crio-487eb714c3165e6f39a46426c35bbbcee1b5e73a5d739fc57067f925c931bd2b WatchSource:0}: Error finding container 487eb714c3165e6f39a46426c35bbbcee1b5e73a5d739fc57067f925c931bd2b: Status 404 returned error can't find the container with id 487eb714c3165e6f39a46426c35bbbcee1b5e73a5d739fc57067f925c931bd2b
Sep 10 12:09:51.488324 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.488301    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17141246_0ee1_4172_8147_f2ace482de53.slice/crio-0075fea141897ea983261cbd29e834d66764abcf3cf28dcdcf1b78789d5f02bc WatchSource:0}: Error finding container 0075fea141897ea983261cbd29e834d66764abcf3cf28dcdcf1b78789d5f02bc: Status 404 returned error can't find the container with id 0075fea141897ea983261cbd29e834d66764abcf3cf28dcdcf1b78789d5f02bc
Sep 10 12:09:51.491751 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.491727    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06141519_5f08_4c52_80bb_1564b403251b.slice/crio-8bc1ca21c0c607f99587b440a62e2cbdb732feec6247a2126a3435bb9c007982 WatchSource:0}: Error finding container 8bc1ca21c0c607f99587b440a62e2cbdb732feec6247a2126a3435bb9c007982: Status 404 returned error can't find the container with id 8bc1ca21c0c607f99587b440a62e2cbdb732feec6247a2126a3435bb9c007982
Sep 10 12:09:51.492637 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.492596    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-rfpvc"
Sep 10 12:09:51.495687 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.495659    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf07e4783_bae9_46c7_88c8_4e8ad5c0b37e.slice/crio-99ebe390228395716285d50465b71a7db239a38cf77a601414feed467e356c92 WatchSource:0}: Error finding container 99ebe390228395716285d50465b71a7db239a38cf77a601414feed467e356c92: Status 404 returned error can't find the container with id 99ebe390228395716285d50465b71a7db239a38cf77a601414feed467e356c92
Sep 10 12:09:51.501443 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.501425    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d"
Sep 10 12:09:51.502890 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.502863    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbfe1596_ceaa_4f73_aa21_59bb62bf1527.slice/crio-94972ad09d1a06ad68908c91b419d1ff8b1d77e2275873455f79694cfed82784 WatchSource:0}: Error finding container 94972ad09d1a06ad68908c91b419d1ff8b1d77e2275873455f79694cfed82784: Status 404 returned error can't find the container with id 94972ad09d1a06ad68908c91b419d1ff8b1d77e2275873455f79694cfed82784
Sep 10 12:09:51.507918 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.507898    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-f8nkn"
Sep 10 12:09:51.515678 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.515656    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-mgklt"
Sep 10 12:09:51.522931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.522904    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:09:51.541553 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.541503    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode071ba5f_61f8_4bde_9a1a_f41d19b319c8.slice/crio-7beba069acfc72a1b3ffdeb32f189efe75dd4efebc7acba33b80b5031543e1ed WatchSource:0}: Error finding container 7beba069acfc72a1b3ffdeb32f189efe75dd4efebc7acba33b80b5031543e1ed: Status 404 returned error can't find the container with id 7beba069acfc72a1b3ffdeb32f189efe75dd4efebc7acba33b80b5031543e1ed
Sep 10 12:09:51.542935 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.542912    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0401f68_2b5d_4c35_bcdd_8cb08eb8a67f.slice/crio-8414f01928feeb4d650d72f67c60082a84a6eaa47b3e1ed5edd8f7b26faafac8 WatchSource:0}: Error finding container 8414f01928feeb4d650d72f67c60082a84a6eaa47b3e1ed5edd8f7b26faafac8: Status 404 returned error can't find the container with id 8414f01928feeb4d650d72f67c60082a84a6eaa47b3e1ed5edd8f7b26faafac8
Sep 10 12:09:51.543908 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.543885    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod461c173a_3009_4c70_a4c1_ba0ca3e8b945.slice/crio-4f578d75d3267868513aad6a850a00955d2ee3439829a448cd1dd82444f76a1c WatchSource:0}: Error finding container 4f578d75d3267868513aad6a850a00955d2ee3439829a448cd1dd82444f76a1c: Status 404 returned error can't find the container with id 4f578d75d3267868513aad6a850a00955d2ee3439829a448cd1dd82444f76a1c
Sep 10 12:09:51.545980 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:09:51.545238    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1b5a6d7_3ac8_400a_ac94_c332aed0ee59.slice/crio-2963f243a4a05880ecb6d2def6d1d9528f4bfef3e2ceab0e59655045e872059e WatchSource:0}: Error finding container 2963f243a4a05880ecb6d2def6d1d9528f4bfef3e2ceab0e59655045e872059e: Status 404 returned error can't find the container with id 2963f243a4a05880ecb6d2def6d1d9528f4bfef3e2ceab0e59655045e872059e
Sep 10 12:09:51.860951 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.860915    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:51.861080 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.860971    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:51.861150 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.861077    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:51.861150 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.861100    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:51.861150 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.861147    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret podName:c982e9cd-3e88-4e3e-9d59-4bbca5053412 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:52.86112928 +0000 UTC m=+4.057644979 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret") pod "global-pull-secret-syncer-w7g5k" (UID: "c982e9cd-3e88-4e3e-9d59-4bbca5053412") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:51.861272 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.861164    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs podName:2948f2cc-e57c-48e7-9f5b-c4efd28fd928 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:52.86115602 +0000 UTC m=+4.057671698 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs") pod "network-metrics-daemon-cxj2p" (UID: "2948f2cc-e57c-48e7-9f5b-c4efd28fd928") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:51.961826 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:51.961796    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:51.961978 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.961959    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:09:51.962059 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.961982    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:09:51.962059 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.961992    2552 projected.go:196] Error preparing data for projected volume kube-api-access-2jgnh for pod openshift-network-diagnostics/network-check-target-hdnv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:51.962059 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:51.962048    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh podName:ee0458f0-6820-424d-b378-1424b0bb7e24 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:52.962031115 +0000 UTC m=+4.158546799 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jgnh" (UniqueName: "kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh") pod "network-check-target-hdnv2" (UID: "ee0458f0-6820-424d-b378-1424b0bb7e24") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:52.247469 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.247370    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:04:51 +0000 UTC" deadline="2028-05-19 15:11:02.712078002 +0000 UTC"
Sep 10 12:09:52.247469 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.247417    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14811h1m10.464665611s"
Sep 10 12:09:52.250420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.249561    2552 generic.go:356] "Generic (PLEG): container finished" podID="e98781e10c3db8f0fd9a7eebd64a1c91" containerID="a195aaa328860ba7aa0a426c9936bc8ec37150fecb82318dd82d95f660e0719a" exitCode=0
Sep 10 12:09:52.250420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.249650    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal" event={"ID":"e98781e10c3db8f0fd9a7eebd64a1c91","Type":"ContainerDied","Data":"a195aaa328860ba7aa0a426c9936bc8ec37150fecb82318dd82d95f660e0719a"}
Sep 10 12:09:52.253036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.252978    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8nkn" event={"ID":"461c173a-3009-4c70-a4c1-ba0ca3e8b945","Type":"ContainerStarted","Data":"4f578d75d3267868513aad6a850a00955d2ee3439829a448cd1dd82444f76a1c"}
Sep 10 12:09:52.256156 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.256132    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d" event={"ID":"e071ba5f-61f8-4bde-9a1a-f41d19b319c8","Type":"ContainerStarted","Data":"7beba069acfc72a1b3ffdeb32f189efe75dd4efebc7acba33b80b5031543e1ed"}
Sep 10 12:09:52.258418 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.258387    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2b6g" event={"ID":"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e","Type":"ContainerStarted","Data":"99ebe390228395716285d50465b71a7db239a38cf77a601414feed467e356c92"}
Sep 10 12:09:52.260169 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.260139    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrmhh" event={"ID":"06141519-5f08-4c52-80bb-1564b403251b","Type":"ContainerStarted","Data":"8bc1ca21c0c607f99587b440a62e2cbdb732feec6247a2126a3435bb9c007982"}
Sep 10 12:09:52.261972 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.261947    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdnqz" event={"ID":"e96c83bb-d317-452c-8326-0e996ab29ebf","Type":"ContainerStarted","Data":"487eb714c3165e6f39a46426c35bbbcee1b5e73a5d739fc57067f925c931bd2b"}
Sep 10 12:09:52.263826 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.263802    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"2963f243a4a05880ecb6d2def6d1d9528f4bfef3e2ceab0e59655045e872059e"}
Sep 10 12:09:52.266132 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.266107    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mgklt" event={"ID":"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f","Type":"ContainerStarted","Data":"8414f01928feeb4d650d72f67c60082a84a6eaa47b3e1ed5edd8f7b26faafac8"}
Sep 10 12:09:52.268150 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.268126    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfpvc" event={"ID":"bbfe1596-ceaa-4f73-aa21-59bb62bf1527","Type":"ContainerStarted","Data":"94972ad09d1a06ad68908c91b419d1ff8b1d77e2275873455f79694cfed82784"}
Sep 10 12:09:52.269862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.269839    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerStarted","Data":"0075fea141897ea983261cbd29e834d66764abcf3cf28dcdcf1b78789d5f02bc"}
Sep 10 12:09:52.870957 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.870871    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:52.870957 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.870930    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:52.871174 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.871103    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:52.871174 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.871168    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs podName:2948f2cc-e57c-48e7-9f5b-c4efd28fd928 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:54.87114811 +0000 UTC m=+6.067663809 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs") pod "network-metrics-daemon-cxj2p" (UID: "2948f2cc-e57c-48e7-9f5b-c4efd28fd928") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:52.871629 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.871607    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:52.871750 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.871667    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret podName:c982e9cd-3e88-4e3e-9d59-4bbca5053412 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:54.871651225 +0000 UTC m=+6.068166922 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret") pod "global-pull-secret-syncer-w7g5k" (UID: "c982e9cd-3e88-4e3e-9d59-4bbca5053412") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:52.971990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:52.971344    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:52.971990 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.971540    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:09:52.971990 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.971565    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:09:52.971990 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.971580    2552 projected.go:196] Error preparing data for projected volume kube-api-access-2jgnh for pod openshift-network-diagnostics/network-check-target-hdnv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:52.971990 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:52.971638    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh podName:ee0458f0-6820-424d-b378-1424b0bb7e24 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:54.97161952 +0000 UTC m=+6.168135205 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jgnh" (UniqueName: "kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh") pod "network-check-target-hdnv2" (UID: "ee0458f0-6820-424d-b378-1424b0bb7e24") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:53.237353 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.237189    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:53.237353 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.237211    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:53.237948 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:53.237348    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:09:53.237948 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.237528    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:53.237948 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:53.237648    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:09:53.237948 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:53.237733    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:09:53.247772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.247713    2552 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 12:04:51 +0000 UTC" deadline="2028-06-18 13:28:08.89007969 +0000 UTC"
Sep 10 12:09:53.247772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.247771    2552 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15529h18m15.642313128s"
Sep 10 12:09:53.290889 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.290848    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal" event={"ID":"e98781e10c3db8f0fd9a7eebd64a1c91","Type":"ContainerStarted","Data":"24c25b6cb6e60fb95a55573f18ad90f4bac436a881cdd16753b092c8988793dc"}
Sep 10 12:09:53.312766 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.312707    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-68.ec2.internal" podStartSLOduration=3.3126858439999998 podStartE2EDuration="3.312685844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:09:53.312306397 +0000 UTC m=+4.508822102" watchObservedRunningTime="2026-09-10 12:09:53.312685844 +0000 UTC m=+4.509201553"
Sep 10 12:09:53.320004 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:53.319967    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal" event={"ID":"72d08dde1ef99723fcb5d6e25dbc85b3","Type":"ContainerStarted","Data":"0a22940eb8a2f840bbfac0b18626502a0cf25a43cf6e2e116d0cdcc92359abc3"}
Sep 10 12:09:54.887996 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:54.887721    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:54.888467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:54.888031    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:54.888467 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.888207    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:54.888467 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.888270    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs podName:2948f2cc-e57c-48e7-9f5b-c4efd28fd928 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:58.888250941 +0000 UTC m=+10.084766637 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs") pod "network-metrics-daemon-cxj2p" (UID: "2948f2cc-e57c-48e7-9f5b-c4efd28fd928") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:54.888467 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.888346    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:54.888467 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.888382    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret podName:c982e9cd-3e88-4e3e-9d59-4bbca5053412 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:58.888369794 +0000 UTC m=+10.084885481 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret") pod "global-pull-secret-syncer-w7g5k" (UID: "c982e9cd-3e88-4e3e-9d59-4bbca5053412") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:54.989910 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:54.989289    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:54.989910 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.989443    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:09:54.989910 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.989475    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:09:54.989910 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.989485    2552 projected.go:196] Error preparing data for projected volume kube-api-access-2jgnh for pod openshift-network-diagnostics/network-check-target-hdnv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:54.989910 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:54.989541    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh podName:ee0458f0-6820-424d-b378-1424b0bb7e24 nodeName:}" failed. No retries permitted until 2026-09-10 12:09:58.989520584 +0000 UTC m=+10.186036267 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jgnh" (UniqueName: "kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh") pod "network-check-target-hdnv2" (UID: "ee0458f0-6820-424d-b378-1424b0bb7e24") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:55.239142 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:55.239053    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:55.239315 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:55.239197    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:09:55.239315 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:55.239262    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:55.239445 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:55.239327    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:09:55.239445 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:55.239372    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:55.239587 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:55.239443    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:09:57.237196 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:57.237158    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:57.237650 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:57.237298    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:09:57.237650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:57.237162    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:57.237650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:57.237354    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:57.237650 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:57.237414    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:09:57.237650 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:57.237490    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:09:58.922652 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:58.922609    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:58.923100 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:58.922670    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:58.923100 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:58.922840    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:58.923100 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:58.922901    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs podName:2948f2cc-e57c-48e7-9f5b-c4efd28fd928 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:06.922881891 +0000 UTC m=+18.119397575 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs") pod "network-metrics-daemon-cxj2p" (UID: "2948f2cc-e57c-48e7-9f5b-c4efd28fd928") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:09:58.923287 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:58.923275    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:58.923340 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:58.923323    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret podName:c982e9cd-3e88-4e3e-9d59-4bbca5053412 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:06.923310724 +0000 UTC m=+18.119826402 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret") pod "global-pull-secret-syncer-w7g5k" (UID: "c982e9cd-3e88-4e3e-9d59-4bbca5053412") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:09:59.023993 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:59.023920    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:59.024165 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:59.024091    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:09:59.024165 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:59.024111    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:09:59.024165 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:59.024122    2552 projected.go:196] Error preparing data for projected volume kube-api-access-2jgnh for pod openshift-network-diagnostics/network-check-target-hdnv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:59.024364 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:59.024183    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh podName:ee0458f0-6820-424d-b378-1424b0bb7e24 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:07.024164601 +0000 UTC m=+18.220680295 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jgnh" (UniqueName: "kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh") pod "network-check-target-hdnv2" (UID: "ee0458f0-6820-424d-b378-1424b0bb7e24") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:09:59.238056 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:59.237522    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:09:59.238056 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:59.237553    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:09:59.238056 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:59.237648    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:09:59.238056 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:09:59.237677    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:09:59.238056 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:59.237751    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:09:59.238056 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:09:59.237890    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:01.237243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:01.237205    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:01.237243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:01.237239    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:01.237760 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:01.237205    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:01.237760 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:01.237357    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:01.237760 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:01.237412    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:01.237760 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:01.237540    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:03.236756 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:03.236723    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:03.237202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:03.236731    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:03.237202 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:03.236872    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:03.237202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:03.236736    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:03.237202 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:03.236915    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:03.237202 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:03.237006    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:05.236804 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:05.236768    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:05.236804 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:05.236791    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:05.237326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:05.236810    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:05.237326 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:05.236910    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:05.237326 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:05.237020    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:05.237326 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:05.237056    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:06.981995 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:06.981962    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:06.982549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:06.982011    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:06.982549 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:06.982137    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:10:06.982549 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:06.982137    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:10:06.982549 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:06.982219    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs podName:2948f2cc-e57c-48e7-9f5b-c4efd28fd928 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:22.982181834 +0000 UTC m=+34.178697513 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs") pod "network-metrics-daemon-cxj2p" (UID: "2948f2cc-e57c-48e7-9f5b-c4efd28fd928") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:10:06.982549 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:06.982236    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret podName:c982e9cd-3e88-4e3e-9d59-4bbca5053412 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:22.982228475 +0000 UTC m=+34.178744152 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret") pod "global-pull-secret-syncer-w7g5k" (UID: "c982e9cd-3e88-4e3e-9d59-4bbca5053412") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:10:07.082849 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:07.082812    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:07.083017 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:07.082993    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:10:07.083068 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:07.083020    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:10:07.083068 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:07.083030    2552 projected.go:196] Error preparing data for projected volume kube-api-access-2jgnh for pod openshift-network-diagnostics/network-check-target-hdnv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:10:07.083148 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:07.083086    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh podName:ee0458f0-6820-424d-b378-1424b0bb7e24 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:23.083072643 +0000 UTC m=+34.279588321 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jgnh" (UniqueName: "kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh") pod "network-check-target-hdnv2" (UID: "ee0458f0-6820-424d-b378-1424b0bb7e24") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:10:07.236311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:07.236220    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:07.236311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:07.236271    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:07.236556 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:07.236224    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:07.236556 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:07.236371    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:07.236556 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:07.236474    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:07.236684 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:07.236554    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:09.237337 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:09.237300    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:09.237813 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:09.237385    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:09.237813 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:09.237406    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:09.237813 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:09.237507    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:09.237813 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:09.237529    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:09.237813 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:09.237649    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:11.236850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.236482    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:11.237891 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.236491    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:11.237891 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.236482    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:11.237891 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:11.236982    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:11.237891 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:11.237121    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:11.237891 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:11.237229    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:11.363171 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.363082    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d" event={"ID":"e071ba5f-61f8-4bde-9a1a-f41d19b319c8","Type":"ContainerStarted","Data":"5a3cba92e23fa3ba937d5e7a794441a52340c6e5c40f0eb3660eb9bd15c868e2"}
Sep 10 12:10:11.365273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.364938    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-f2b6g" event={"ID":"f07e4783-bae9-46c7-88c8-4e8ad5c0b37e","Type":"ContainerStarted","Data":"2b4c546643ffaafab962a2aee84e5d301e57814b97338fbdc3b8c616c620ae8f"}
Sep 10 12:10:11.366593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.366556    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-gdnqz" event={"ID":"e96c83bb-d317-452c-8326-0e996ab29ebf","Type":"ContainerStarted","Data":"dd689b40c57773bf8cdafd4b248cb70a0a9924a5f2efb7fd1b7ebd9501927cdd"}
Sep 10 12:10:11.370193 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.370163    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"a76225073b32fbafb6e61b6368ee459b7b7b243ca41c7eb9bc3048f346a30d3a"}
Sep 10 12:10:11.370348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.370251    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"5db7d3b4a0051194d3ead862e6c84e6e6eb11c0b9d71f6eb8617ee25eda7d970"}
Sep 10 12:10:11.370348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.370289    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"159d553ec0eb2a7026e7cac8983a1d2f02c3a8056f91d94b5c4c988355d007eb"}
Sep 10 12:10:11.370348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.370335    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"7113d7e34113bacf52f5ae0dab504a9718b8b3ce868dc28b67796cd862c6d9f9"}
Sep 10 12:10:11.370549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.370377    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"b2d6877e3eaf50663ef32eb9810e197a0831a5d9a6c199cc7b0a634e99f17aea"}
Sep 10 12:10:11.370549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.370405    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"4d1b4a1ad80ae68838ecf1e42a90b5fb692b19df95173614276154e4d63869ce"}
Sep 10 12:10:11.372066 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.372040    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-mgklt" event={"ID":"e0401f68-2b5d-4c35-bcdd-8cb08eb8a67f","Type":"ContainerStarted","Data":"5d4b3c9b035403c90159d7ea89b94e85009cb4e8ce6b3d7985d2a66c72a0f9cf"}
Sep 10 12:10:11.373889 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.373864    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-rfpvc" event={"ID":"bbfe1596-ceaa-4f73-aa21-59bb62bf1527","Type":"ContainerStarted","Data":"d1745babda764bb52f8075b127300c3d0965ca850fae6c79bb9740e229b80390"}
Sep 10 12:10:11.375718 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.375688    2552 generic.go:356] "Generic (PLEG): container finished" podID="17141246-0ee1-4172-8147-f2ace482de53" containerID="d0e513082df696a45f91398a5b74468865be69b01b80a3d3ada9f392ffade2a0" exitCode=0
Sep 10 12:10:11.375847 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.375797    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerDied","Data":"d0e513082df696a45f91398a5b74468865be69b01b80a3d3ada9f392ffade2a0"}
Sep 10 12:10:11.377929 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.377905    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-f8nkn" event={"ID":"461c173a-3009-4c70-a4c1-ba0ca3e8b945","Type":"ContainerStarted","Data":"32bf4e3b51984f4d8322ec789d444a8fa964ad1f8fb0756ea13d90d29d0513b8"}
Sep 10 12:10:11.389853 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.389783    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-68.ec2.internal" podStartSLOduration=21.389757556 podStartE2EDuration="21.389757556s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:09:53.346319249 +0000 UTC m=+4.542834953" watchObservedRunningTime="2026-09-10 12:10:11.389757556 +0000 UTC m=+22.586273258"
Sep 10 12:10:11.390179 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.390150    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-f2b6g" podStartSLOduration=3.5544899389999998 podStartE2EDuration="22.390140211s" totalImagesPullingTime="18.835650272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:11.389728328 +0000 UTC m=+22.586244027" watchObservedRunningTime="2026-09-10 12:10:11.390140211 +0000 UTC m=+22.586655913"
Sep 10 12:10:11.408146 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.408092    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-mgklt" podStartSLOduration=3.5895537060000002 podStartE2EDuration="22.408072736s" totalImagesPullingTime="18.81851903s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:11.407393943 +0000 UTC m=+22.603909644" watchObservedRunningTime="2026-09-10 12:10:11.408072736 +0000 UTC m=+22.604588436"
Sep 10 12:10:11.452949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.452872    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-rfpvc" podStartSLOduration=3.772980613 podStartE2EDuration="22.45285066s" totalImagesPullingTime="18.679870047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:11.424383113 +0000 UTC m=+22.620898815" watchObservedRunningTime="2026-09-10 12:10:11.45285066 +0000 UTC m=+22.649366361"
Sep 10 12:10:11.467142 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.467082    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-gdnqz" podStartSLOduration=3.651611666 podStartE2EDuration="22.467063631s" totalImagesPullingTime="18.815451965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:11.466683772 +0000 UTC m=+22.663199474" watchObservedRunningTime="2026-09-10 12:10:11.467063631 +0000 UTC m=+22.663579333"
Sep 10 12:10:11.481779 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:11.481718    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-f8nkn" podStartSLOduration=3.695800749 podStartE2EDuration="22.481694309s" totalImagesPullingTime="18.78589356s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:11.481155711 +0000 UTC m=+22.677671412" watchObservedRunningTime="2026-09-10 12:10:11.481694309 +0000 UTC m=+22.678210011"
Sep 10 12:10:12.235133 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:12.234973    2552 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 12:10:12.382161 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:12.382124    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d" event={"ID":"e071ba5f-61f8-4bde-9a1a-f41d19b319c8","Type":"ContainerStarted","Data":"b3e2793fb7512328f049f488fae3e2c4fe1c7dc8010126960be2ee9049a9bfa7"}
Sep 10 12:10:12.384063 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:12.384021    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-rrmhh" event={"ID":"06141519-5f08-4c52-80bb-1564b403251b","Type":"ContainerStarted","Data":"e12ac3749374af40ad0dae361305df42131dfa978eaf5ce7b2a3227adeab26f1"}
Sep 10 12:10:12.406975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:12.406907    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-rrmhh" podStartSLOduration=4.568984958 podStartE2EDuration="23.406882753s" totalImagesPullingTime="18.837897795s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:12.406150299 +0000 UTC m=+23.602665999" watchObservedRunningTime="2026-09-10 12:10:12.406882753 +0000 UTC m=+23.603398455"
Sep 10 12:10:12.845404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:12.845312    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:10:12.846108 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:12.846084    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:10:13.215208 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.215091    2552 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T12:10:12.23512525Z","UUID":"5136ed5d-f87f-4bd5-a065-0a0ef1879813","Handler":null,"Name":"","Endpoint":""}
Sep 10 12:10:13.217467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.217077    2552 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 12:10:13.217467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.217114    2552 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 12:10:13.238895 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.238859    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:13.239062 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:13.238977    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:13.239267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.239247    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:13.239379 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:13.239358    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:13.239471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.239451    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:13.239576 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:13.239556    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:13.387417 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.387384    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:10:13.387899 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:13.387877    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-gdnqz"
Sep 10 12:10:14.392759 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:14.392710    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d" event={"ID":"e071ba5f-61f8-4bde-9a1a-f41d19b319c8","Type":"ContainerStarted","Data":"8b6e1c64bbcae096ba940da75cd4faaecd0379542523895bfd183749a98b8793"}
Sep 10 12:10:14.396916 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:14.396880    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"2142e6c466a0c80c82b31b869c6d57235622543711f1f062fb302f49c81d5cf9"}
Sep 10 12:10:14.411746 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:14.411686    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-vxv5d" podStartSLOduration=3.719448816 podStartE2EDuration="25.411662406s" totalImagesPullingTime="21.69221359s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:14.411634972 +0000 UTC m=+25.608150673" watchObservedRunningTime="2026-09-10 12:10:14.411662406 +0000 UTC m=+25.608178107"
Sep 10 12:10:15.236898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:15.236369    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:15.236898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:15.236383    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:15.236898 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:15.236522    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:15.236898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:15.236383    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:15.236898 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:15.236568    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:15.236898 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:15.236648    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:16.405258 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.404964    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" event={"ID":"f1b5a6d7-3ac8-400a-ac94-c332aed0ee59","Type":"ContainerStarted","Data":"04116c79cf8203b6e73ae217d807613773505e65df93c425dcc30741d39779ca"}
Sep 10 12:10:16.405963 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.405323    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:10:16.405963 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.405398    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:10:16.405963 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.405461    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:10:16.407084 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.407055    2552 generic.go:356] "Generic (PLEG): container finished" podID="17141246-0ee1-4172-8147-f2ace482de53" containerID="4f605eaa37c4e5e4531fa4c2bdecb0236fdf7bc6ba22ebed98797e03afd4972b" exitCode=0
Sep 10 12:10:16.407189 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.407121    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerDied","Data":"4f605eaa37c4e5e4531fa4c2bdecb0236fdf7bc6ba22ebed98797e03afd4972b"}
Sep 10 12:10:16.423226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.423200    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:10:16.423363 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.423353    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:10:16.441761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.441708    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45" podStartSLOduration=7.401305191 podStartE2EDuration="26.44169158s" totalImagesPullingTime="19.040386389s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:16.441355528 +0000 UTC m=+27.637871228" watchObservedRunningTime="2026-09-10 12:10:16.44169158 +0000 UTC m=+27.638207282"
Sep 10 12:10:16.903165 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:16.903133    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfpvc_bbfe1596-ceaa-4f73-aa21-59bb62bf1527/dns-node-resolver/0.log"
Sep 10 12:10:17.236784 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:17.236692    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:17.236784 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:17.236728    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:17.236961 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:17.236692    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:17.236961 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:17.236816    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:17.236961 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:17.236928    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:17.237066 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:17.237027    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:17.638026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:17.637999    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8nkn_461c173a-3009-4c70-a4c1-ba0ca3e8b945/node-ca/0.log"
Sep 10 12:10:18.325598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.325380    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cxj2p"]
Sep 10 12:10:18.325775 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.325671    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:18.325775 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:18.325760    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:18.340895 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.340861    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hdnv2"]
Sep 10 12:10:18.341054 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.340978    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:18.341124 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:18.341081    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:18.341773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.341744    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w7g5k"]
Sep 10 12:10:18.341908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.341832    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:18.341999 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:18.341979    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:18.416008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.415972    2552 generic.go:356] "Generic (PLEG): container finished" podID="17141246-0ee1-4172-8147-f2ace482de53" containerID="f0a1a69e0cfa6f50cf4d2379509b0320bc8529fb127e7ae7d02ae4aa56dcdf83" exitCode=0
Sep 10 12:10:18.416208 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:18.416019    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerDied","Data":"f0a1a69e0cfa6f50cf4d2379509b0320bc8529fb127e7ae7d02ae4aa56dcdf83"}
Sep 10 12:10:19.421033 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:19.420941    2552 generic.go:356] "Generic (PLEG): container finished" podID="17141246-0ee1-4172-8147-f2ace482de53" containerID="8f6c5d525a046ca76ba85a8c929551fe4b371739ca4e0f9c49ae65e72c669354" exitCode=0
Sep 10 12:10:19.421033 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:19.421011    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerDied","Data":"8f6c5d525a046ca76ba85a8c929551fe4b371739ca4e0f9c49ae65e72c669354"}
Sep 10 12:10:20.236307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:20.236268    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:20.236307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:20.236288    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:20.236585 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:20.236403    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:20.236585 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:20.236511    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:20.236585 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:20.236582    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:20.236773 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:20.236665    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:22.236199 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:22.236166    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:22.236199 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:22.236185    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:22.236761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:22.236180    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:22.236761 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:22.236299    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:22.236761 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:22.236400    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:22.236761 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:22.236533    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:23.007851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:23.007799    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:23.008122 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:23.007864    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:23.008122 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.007978    2552 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 12:10:23.008122 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.008011    2552 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:10:23.008122 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.008048    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret podName:c982e9cd-3e88-4e3e-9d59-4bbca5053412 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:55.008030679 +0000 UTC m=+66.204546377 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret") pod "global-pull-secret-syncer-w7g5k" (UID: "c982e9cd-3e88-4e3e-9d59-4bbca5053412") : object "kube-system"/"original-pull-secret" not registered
Sep 10 12:10:23.008122 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.008072    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs podName:2948f2cc-e57c-48e7-9f5b-c4efd28fd928 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:55.008057807 +0000 UTC m=+66.204573487 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs") pod "network-metrics-daemon-cxj2p" (UID: "2948f2cc-e57c-48e7-9f5b-c4efd28fd928") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 12:10:23.108901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:23.108861    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:23.109081 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.109031    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 12:10:23.109081 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.109050    2552 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 12:10:23.109081 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.109063    2552 projected.go:196] Error preparing data for projected volume kube-api-access-2jgnh for pod openshift-network-diagnostics/network-check-target-hdnv2: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:10:23.109216 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:23.109126    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh podName:ee0458f0-6820-424d-b378-1424b0bb7e24 nodeName:}" failed. No retries permitted until 2026-09-10 12:10:55.109110936 +0000 UTC m=+66.305626613 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-2jgnh" (UniqueName: "kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh") pod "network-check-target-hdnv2" (UID: "ee0458f0-6820-424d-b378-1424b0bb7e24") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 12:10:24.236631 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:24.236597    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:24.236631 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:24.236635    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:24.237172 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:24.236711    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:24.237172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:24.236754    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:24.237172 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:24.236824    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:24.237172 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:24.236909    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:24.437648 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:24.437610    2552 generic.go:356] "Generic (PLEG): container finished" podID="17141246-0ee1-4172-8147-f2ace482de53" containerID="9cb4eb52052165473412be896b505217c1c09ead0443534db4325ec80c08ca0a" exitCode=0
Sep 10 12:10:24.437828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:24.437678    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerDied","Data":"9cb4eb52052165473412be896b505217c1c09ead0443534db4325ec80c08ca0a"}
Sep 10 12:10:25.443196 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:25.443155    2552 generic.go:356] "Generic (PLEG): container finished" podID="17141246-0ee1-4172-8147-f2ace482de53" containerID="d1b1256b3077a685021acd8dc11643c043f12fbefc597e963ca0075453664697" exitCode=0
Sep 10 12:10:25.443605 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:25.443207    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerDied","Data":"d1b1256b3077a685021acd8dc11643c043f12fbefc597e963ca0075453664697"}
Sep 10 12:10:26.236264 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:26.236229    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:26.236488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:26.236236    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:26.236488 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:26.236360    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:26.236488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:26.236236    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:26.236488 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:26.236472    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:26.236776 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:26.236514    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:26.449017 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:26.448983    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" event={"ID":"17141246-0ee1-4172-8147-f2ace482de53","Type":"ContainerStarted","Data":"c96272ed0362c0358ba112a621599953ce4eef39a001c45c6f40861ff3155cc9"}
Sep 10 12:10:28.236513 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:28.236470    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:28.236513 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:28.236506    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:28.237118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:28.236506    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:28.237118 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:28.236600    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:28.237118 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:28.236663    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:28.237118 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:28.236738    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:30.236262 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:30.236222    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:30.236696 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:30.236326    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:30.236696 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:30.236334    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:30.236696 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:30.236357    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:30.236696 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:30.236449    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:30.236696 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:30.236547    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:32.236946 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:32.236906    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:32.237359 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:32.237012    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:32.237359 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:32.237020    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:32.237359 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:32.237042    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:32.237359 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:32.237119    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:32.237359 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:32.237174    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:34.236410 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:34.236345    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:34.236410 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:34.236345    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:34.237180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:34.236345    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:34.237180 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:34.236586    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:34.237180 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:34.236731    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:34.237180 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:34.236837    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:36.236317 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:36.236282    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:36.236731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:36.236284    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:36.236731 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:36.236397    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:36.236731 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:36.236497    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:36.236731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:36.236291    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:36.236731 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:36.236563    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:38.236797 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:38.236751    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:38.237290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:38.236877    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:38.237290 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:38.236886    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:38.237290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:38.236907    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:38.237290 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:38.236989    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:38.237290 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:38.237069    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:40.236497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:40.236461    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:40.236940 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:40.236504    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:40.236940 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:40.236461    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:40.236940 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:40.236579    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:40.236940 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:40.236655    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:40.236940 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:40.236706    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:42.236591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:42.236555    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:42.237046 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:42.236555    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:42.237046 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:42.236679    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-cxj2p" podUID="2948f2cc-e57c-48e7-9f5b-c4efd28fd928"
Sep 10 12:10:42.237046 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:42.236730    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-hdnv2" podUID="ee0458f0-6820-424d-b378-1424b0bb7e24"
Sep 10 12:10:42.237046 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:42.236555    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:42.237046 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:10:42.236837    2552 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w7g5k" podUID="c982e9cd-3e88-4e3e-9d59-4bbca5053412"
Sep 10 12:10:43.071735 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.071702    2552 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-68.ec2.internal" event="NodeReady"
Sep 10 12:10:43.071910 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.071846    2552 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 12:10:43.133948 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.133885    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-pxwnc" podStartSLOduration=24.259674506 podStartE2EDuration="54.133849198s" totalImagesPullingTime="29.874174692s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:26.490574275 +0000 UTC m=+37.687089976" watchObservedRunningTime="2026-09-10 12:10:43.133849198 +0000 UTC m=+54.330364889"
Sep 10 12:10:43.134913 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.134813    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-dffdcb8ff-rfpg2"]
Sep 10 12:10:43.136883 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.136865    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.139488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.139460    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 12:10:43.139640 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.139623    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 12:10:43.139737 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.139722    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 12:10:43.142421 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.142400    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-7vctr\""
Sep 10 12:10:43.147141 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.147119    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 12:10:43.159605 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.159581    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dffdcb8ff-rfpg2"]
Sep 10 12:10:43.185632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.185543    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-77b6m"]
Sep 10 12:10:43.187586 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.187569    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.190526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.190485    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 12:10:43.190670 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.190554    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 12:10:43.191020 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.190999    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qwz2g\""
Sep 10 12:10:43.191125 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.191065    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 12:10:43.191307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.191286    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 12:10:43.204672 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.204630    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77b6m"]
Sep 10 12:10:43.205283 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.205260    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-cmhtc"]
Sep 10 12:10:43.207644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.207622    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.210539 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.210515    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 12:10:43.210539 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.210529    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-7696c\""
Sep 10 12:10:43.210727 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.210684    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 12:10:43.210806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.210791    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 12:10:43.218836 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.218808    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cmhtc"]
Sep 10 12:10:43.249348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249312    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-trusted-ca\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.249348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249353    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-installation-pull-secrets\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.249796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249377    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-image-registry-private-configuration\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.249796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249466    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m6lkk\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-kube-api-access-m6lkk\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.249796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249499    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-registry-tls\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.249796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249558    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-registry-certificates\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.249796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249574    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-bound-sa-token\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.249796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.249608    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-ca-trust-extracted\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.282152 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.282117    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-xxvfc"]
Sep 10 12:10:43.284204 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.284183    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.288665 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.288637    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 12:10:43.288665 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.288664    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-2gxdf\""
Sep 10 12:10:43.288830 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.288805    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 12:10:43.306059 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.306015    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxvfc"]
Sep 10 12:10:43.350584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350547    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02f33ccf-a68b-4de0-822c-5e710ed402db-cert\") pod \"ingress-canary-cmhtc\" (UID: \"02f33ccf-a68b-4de0-822c-5e710ed402db\") " pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.350763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350591    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m6lkk\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-kube-api-access-m6lkk\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.350763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350641    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-registry-tls\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.350763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350673    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5ncd\" (UniqueName: \"kubernetes.io/projected/c268d932-b3fa-4d93-a76d-3fba80f645e2-kube-api-access-q5ncd\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.350878 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350765    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ph6r\" (UniqueName: \"kubernetes.io/projected/02f33ccf-a68b-4de0-822c-5e710ed402db-kube-api-access-8ph6r\") pod \"ingress-canary-cmhtc\" (UID: \"02f33ccf-a68b-4de0-822c-5e710ed402db\") " pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.350878 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350825    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-registry-certificates\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.350878 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350863    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-bound-sa-token\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.351010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.350892    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c268d932-b3fa-4d93-a76d-3fba80f645e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.351010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351002    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-ca-trust-extracted\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.351112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351042    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-trusted-ca\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.351112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351077    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-installation-pull-secrets\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.351213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351115    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c268d932-b3fa-4d93-a76d-3fba80f645e2-crio-socket\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.351213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351147    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-image-registry-private-configuration\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.351213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351206    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c268d932-b3fa-4d93-a76d-3fba80f645e2-data-volume\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.351358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351234    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c268d932-b3fa-4d93-a76d-3fba80f645e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.351412 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351398    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-ca-trust-extracted\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.351760 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.351741    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-registry-certificates\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.352226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.352205    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-trusted-ca\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.355144 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.355095    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-image-registry-private-configuration\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.355327 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.355303    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-installation-pull-secrets\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.355419 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.355405    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-registry-tls\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.365243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.365205    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-bound-sa-token\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.365395 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.365301    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m6lkk\" (UniqueName: \"kubernetes.io/projected/b9e1f7f5-f5af-426d-a702-d2989ae3c6a1-kube-api-access-m6lkk\") pod \"image-registry-dffdcb8ff-rfpg2\" (UID: \"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1\") " pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.447870 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.447765    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:43.452707 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452677    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-metrics-tls\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.452844 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452721    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c268d932-b3fa-4d93-a76d-3fba80f645e2-data-volume\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.452844 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452745    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c268d932-b3fa-4d93-a76d-3fba80f645e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.452844 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452770    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02f33ccf-a68b-4de0-822c-5e710ed402db-cert\") pod \"ingress-canary-cmhtc\" (UID: \"02f33ccf-a68b-4de0-822c-5e710ed402db\") " pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.452844 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452795    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-config-volume\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.452844 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452821    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-tmp-dir\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.453092 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452847    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5ncd\" (UniqueName: \"kubernetes.io/projected/c268d932-b3fa-4d93-a76d-3fba80f645e2-kube-api-access-q5ncd\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.453092 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452931    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ph6r\" (UniqueName: \"kubernetes.io/projected/02f33ccf-a68b-4de0-822c-5e710ed402db-kube-api-access-8ph6r\") pod \"ingress-canary-cmhtc\" (UID: \"02f33ccf-a68b-4de0-822c-5e710ed402db\") " pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.453092 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.452969    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c268d932-b3fa-4d93-a76d-3fba80f645e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.453331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.453305    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c268d932-b3fa-4d93-a76d-3fba80f645e2-crio-socket\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.453466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.453348    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kn5kx\" (UniqueName: \"kubernetes.io/projected/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-kube-api-access-kn5kx\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.453466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.453354    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c268d932-b3fa-4d93-a76d-3fba80f645e2-data-volume\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.453593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.453537    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c268d932-b3fa-4d93-a76d-3fba80f645e2-crio-socket\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.454419 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.454398    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c268d932-b3fa-4d93-a76d-3fba80f645e2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.455761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.455743    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c268d932-b3fa-4d93-a76d-3fba80f645e2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.455822 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.455791    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/02f33ccf-a68b-4de0-822c-5e710ed402db-cert\") pod \"ingress-canary-cmhtc\" (UID: \"02f33ccf-a68b-4de0-822c-5e710ed402db\") " pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.463932 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.463875    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5ncd\" (UniqueName: \"kubernetes.io/projected/c268d932-b3fa-4d93-a76d-3fba80f645e2-kube-api-access-q5ncd\") pod \"insights-runtime-extractor-77b6m\" (UID: \"c268d932-b3fa-4d93-a76d-3fba80f645e2\") " pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.464212 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.463874    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ph6r\" (UniqueName: \"kubernetes.io/projected/02f33ccf-a68b-4de0-822c-5e710ed402db-kube-api-access-8ph6r\") pod \"ingress-canary-cmhtc\" (UID: \"02f33ccf-a68b-4de0-822c-5e710ed402db\") " pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.497452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.497392    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-77b6m"
Sep 10 12:10:43.517727 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.517695    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-cmhtc"
Sep 10 12:10:43.554273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.554241    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kn5kx\" (UniqueName: \"kubernetes.io/projected/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-kube-api-access-kn5kx\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.554402 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.554348    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-metrics-tls\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.554499 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.554413    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-config-volume\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.554499 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.554486    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-tmp-dir\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.555897 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.555222    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-tmp-dir\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.555897 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.555863    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-config-volume\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.558813 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.558787    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-metrics-tls\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.566933 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.566911    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kn5kx\" (UniqueName: \"kubernetes.io/projected/9e7f3c5e-3819-44f9-b6ff-94f466858b2c-kube-api-access-kn5kx\") pod \"dns-default-xxvfc\" (UID: \"9e7f3c5e-3819-44f9-b6ff-94f466858b2c\") " pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.593101 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.593067    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:43.745695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.745609    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-dffdcb8ff-rfpg2"]
Sep 10 12:10:43.748579 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:10:43.748546    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9e1f7f5_f5af_426d_a702_d2989ae3c6a1.slice/crio-6a16742f665921c83d4536181d300232937adfe9f004735034af3c4afc92e5fd WatchSource:0}: Error finding container 6a16742f665921c83d4536181d300232937adfe9f004735034af3c4afc92e5fd: Status 404 returned error can't find the container with id 6a16742f665921c83d4536181d300232937adfe9f004735034af3c4afc92e5fd
Sep 10 12:10:43.783545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.783235    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-77b6m"]
Sep 10 12:10:43.787486 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:10:43.787455    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc268d932_b3fa_4d93_a76d_3fba80f645e2.slice/crio-a3db81873da63d3584d2fb0c86e616cc37c35371a98beaff7978f3458b6ddc5b WatchSource:0}: Error finding container a3db81873da63d3584d2fb0c86e616cc37c35371a98beaff7978f3458b6ddc5b: Status 404 returned error can't find the container with id a3db81873da63d3584d2fb0c86e616cc37c35371a98beaff7978f3458b6ddc5b
Sep 10 12:10:43.802887 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.802854    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-cmhtc"]
Sep 10 12:10:43.806535 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:10:43.806498    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02f33ccf_a68b_4de0_822c_5e710ed402db.slice/crio-7135d3b4e495820243a963201f074ff89fac08ebc3d8c421bd8b262d4e78bb2d WatchSource:0}: Error finding container 7135d3b4e495820243a963201f074ff89fac08ebc3d8c421bd8b262d4e78bb2d: Status 404 returned error can't find the container with id 7135d3b4e495820243a963201f074ff89fac08ebc3d8c421bd8b262d4e78bb2d
Sep 10 12:10:43.872238 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:43.872207    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-xxvfc"]
Sep 10 12:10:43.875976 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:10:43.875949    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e7f3c5e_3819_44f9_b6ff_94f466858b2c.slice/crio-2476bc6cb1e709f1aa28c7400cdc58a5c9c107ca5120cf81a152753839337825 WatchSource:0}: Error finding container 2476bc6cb1e709f1aa28c7400cdc58a5c9c107ca5120cf81a152753839337825: Status 404 returned error can't find the container with id 2476bc6cb1e709f1aa28c7400cdc58a5c9c107ca5120cf81a152753839337825
Sep 10 12:10:44.237164 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.237126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:44.237364 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.237127    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:44.237364 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.237127    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:44.245856 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.245820    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 12:10:44.247020 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.246937    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 12:10:44.247020 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.246996    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7mt6m\""
Sep 10 12:10:44.247179 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.246945    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 12:10:44.247239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.246945    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hpk75\""
Sep 10 12:10:44.247239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.246938    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 12:10:44.491487 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.491358    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cmhtc" event={"ID":"02f33ccf-a68b-4de0-822c-5e710ed402db","Type":"ContainerStarted","Data":"7135d3b4e495820243a963201f074ff89fac08ebc3d8c421bd8b262d4e78bb2d"}
Sep 10 12:10:44.493165 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.493137    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77b6m" event={"ID":"c268d932-b3fa-4d93-a76d-3fba80f645e2","Type":"ContainerStarted","Data":"d40d41620b46b292494a659f062a502337268aaf5bf183e44d8477aecaa0d721"}
Sep 10 12:10:44.493316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.493188    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77b6m" event={"ID":"c268d932-b3fa-4d93-a76d-3fba80f645e2","Type":"ContainerStarted","Data":"4fba4f17a8dcc562298aafd0df3b06571ed0a5a67b46e223ce4209c5863ba1eb"}
Sep 10 12:10:44.493316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.493218    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77b6m" event={"ID":"c268d932-b3fa-4d93-a76d-3fba80f645e2","Type":"ContainerStarted","Data":"a3db81873da63d3584d2fb0c86e616cc37c35371a98beaff7978f3458b6ddc5b"}
Sep 10 12:10:44.494345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.494314    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxvfc" event={"ID":"9e7f3c5e-3819-44f9-b6ff-94f466858b2c","Type":"ContainerStarted","Data":"2476bc6cb1e709f1aa28c7400cdc58a5c9c107ca5120cf81a152753839337825"}
Sep 10 12:10:44.495850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.495831    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2" event={"ID":"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1","Type":"ContainerStarted","Data":"4b7c73cea1b97a50f83364ae5bbdfd9fa2fc97148889b98c3e043014e6a81a32"}
Sep 10 12:10:44.495939 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.495865    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2" event={"ID":"b9e1f7f5-f5af-426d-a702-d2989ae3c6a1","Type":"ContainerStarted","Data":"6a16742f665921c83d4536181d300232937adfe9f004735034af3c4afc92e5fd"}
Sep 10 12:10:44.495990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.495973    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:10:44.564384 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:44.564324    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2" podStartSLOduration=1.564306804 podStartE2EDuration="1.564306804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:44.562744995 +0000 UTC m=+55.759260694" watchObservedRunningTime="2026-09-10 12:10:44.564306804 +0000 UTC m=+55.760822504"
Sep 10 12:10:48.428310 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.428276    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-mhb45"
Sep 10 12:10:48.509204 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.509105    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-cmhtc" event={"ID":"02f33ccf-a68b-4de0-822c-5e710ed402db","Type":"ContainerStarted","Data":"f22f16b61011d5ca059624b6770051cdc3e37856530287e7c2e4f955b8e96d7e"}
Sep 10 12:10:48.511378 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.511348    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-77b6m" event={"ID":"c268d932-b3fa-4d93-a76d-3fba80f645e2","Type":"ContainerStarted","Data":"2be756bd96124a3d5536a5d2e9c081e377e3b1a2158358468f2ca3c6a5863071"}
Sep 10 12:10:48.513075 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.513046    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxvfc" event={"ID":"9e7f3c5e-3819-44f9-b6ff-94f466858b2c","Type":"ContainerStarted","Data":"9e98b724f0511fd80c78ed14660c92015fa0d065d5fedc458a6870f754d9e5c7"}
Sep 10 12:10:48.513214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.513100    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-xxvfc" event={"ID":"9e7f3c5e-3819-44f9-b6ff-94f466858b2c","Type":"ContainerStarted","Data":"d596a19e7167ff5e930100c78cc8efb7ab1bc442964351c41ea3a391f016abb7"}
Sep 10 12:10:48.513214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.513140    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:10:48.527590 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.527540    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-cmhtc" podStartSLOduration=1.453101185 podStartE2EDuration="5.527522s" totalImagesPullingTime="4.074420815s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:48.527027628 +0000 UTC m=+59.723543356" watchObservedRunningTime="2026-09-10 12:10:48.527522 +0000 UTC m=+59.724037700"
Sep 10 12:10:48.550200 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.550144    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-77b6m" podStartSLOduration=1.573666035 podStartE2EDuration="5.550125478s" totalImagesPullingTime="3.976459443s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:10:43 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:48.548925448 +0000 UTC m=+59.745441150" watchObservedRunningTime="2026-09-10 12:10:48.550125478 +0000 UTC m=+59.746641172"
Sep 10 12:10:48.567967 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:48.567903    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-xxvfc" podStartSLOduration=1.57033837 podStartE2EDuration="5.567884968s" totalImagesPullingTime="3.997546598s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:10:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:48.566682424 +0000 UTC m=+59.763198125" watchObservedRunningTime="2026-09-10 12:10:48.567884968 +0000 UTC m=+59.764400668"
Sep 10 12:10:55.046901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.046852    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:55.046901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.046903    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:55.049681 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.049661    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 12:10:55.050490 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.050473    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 12:10:55.059874 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.059842    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2948f2cc-e57c-48e7-9f5b-c4efd28fd928-metrics-certs\") pod \"network-metrics-daemon-cxj2p\" (UID: \"2948f2cc-e57c-48e7-9f5b-c4efd28fd928\") " pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:55.059975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.059922    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c982e9cd-3e88-4e3e-9d59-4bbca5053412-original-pull-secret\") pod \"global-pull-secret-syncer-w7g5k\" (UID: \"c982e9cd-3e88-4e3e-9d59-4bbca5053412\") " pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:55.064864 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.064842    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w7g5k"
Sep 10 12:10:55.148111 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.148073    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:55.150810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.150785    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 12:10:55.161547 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.161520    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 12:10:55.173406 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.173322    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2jgnh\" (UniqueName: \"kubernetes.io/projected/ee0458f0-6820-424d-b378-1424b0bb7e24-kube-api-access-2jgnh\") pod \"network-check-target-hdnv2\" (UID: \"ee0458f0-6820-424d-b378-1424b0bb7e24\") " pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:55.353315 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.353243    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-7mt6m\""
Sep 10 12:10:55.360519 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.360497    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-hpk75\""
Sep 10 12:10:55.361511 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.361494    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-cxj2p"
Sep 10 12:10:55.367346 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.367320    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w7g5k"]
Sep 10 12:10:55.369408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.369381    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:10:55.370415 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:10:55.370383    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc982e9cd_3e88_4e3e_9d59_4bbca5053412.slice/crio-145c7b44cf45fe58ff3b659330198a80a360ead1d98a07d3f2833ef01ffda01b WatchSource:0}: Error finding container 145c7b44cf45fe58ff3b659330198a80a360ead1d98a07d3f2833ef01ffda01b: Status 404 returned error can't find the container with id 145c7b44cf45fe58ff3b659330198a80a360ead1d98a07d3f2833ef01ffda01b
Sep 10 12:10:55.536019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.535987    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w7g5k" event={"ID":"c982e9cd-3e88-4e3e-9d59-4bbca5053412","Type":"ContainerStarted","Data":"145c7b44cf45fe58ff3b659330198a80a360ead1d98a07d3f2833ef01ffda01b"}
Sep 10 12:10:55.637159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.636996    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-cxj2p"]
Sep 10 12:10:55.639643 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:10:55.639620    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2948f2cc_e57c_48e7_9f5b_c4efd28fd928.slice/crio-8a4e048b1df3156c7fade2f659a81a139126db220faf75a72f9142a118be5f03 WatchSource:0}: Error finding container 8a4e048b1df3156c7fade2f659a81a139126db220faf75a72f9142a118be5f03: Status 404 returned error can't find the container with id 8a4e048b1df3156c7fade2f659a81a139126db220faf75a72f9142a118be5f03
Sep 10 12:10:55.657262 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:55.657237    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-hdnv2"]
Sep 10 12:10:55.661593 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:10:55.661461    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0458f0_6820_424d_b378_1424b0bb7e24.slice/crio-81f694fc00ee59ba555ea715c64891fb8950e22e3cbc40f2c9a68030d5908446 WatchSource:0}: Error finding container 81f694fc00ee59ba555ea715c64891fb8950e22e3cbc40f2c9a68030d5908446: Status 404 returned error can't find the container with id 81f694fc00ee59ba555ea715c64891fb8950e22e3cbc40f2c9a68030d5908446
Sep 10 12:10:56.542894 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:56.542783    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cxj2p" event={"ID":"2948f2cc-e57c-48e7-9f5b-c4efd28fd928","Type":"ContainerStarted","Data":"8a4e048b1df3156c7fade2f659a81a139126db220faf75a72f9142a118be5f03"}
Sep 10 12:10:56.545521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:56.545476    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hdnv2" event={"ID":"ee0458f0-6820-424d-b378-1424b0bb7e24","Type":"ContainerStarted","Data":"81f694fc00ee59ba555ea715c64891fb8950e22e3cbc40f2c9a68030d5908446"}
Sep 10 12:10:57.551349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:57.551209    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cxj2p" event={"ID":"2948f2cc-e57c-48e7-9f5b-c4efd28fd928","Type":"ContainerStarted","Data":"1d7283606b9e38994082b5f2f5efbc938f1ff578ff866bd8673e6d20789308db"}
Sep 10 12:10:57.551349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:57.551263    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-cxj2p" event={"ID":"2948f2cc-e57c-48e7-9f5b-c4efd28fd928","Type":"ContainerStarted","Data":"343a1a274505e96e29f798b41b3973ecdfc79295f8cb8e9ae8bbea52c844c8c2"}
Sep 10 12:10:57.571286 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:57.571240    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-cxj2p" podStartSLOduration=67.06021046 podStartE2EDuration="1m8.571223505s" totalImagesPullingTime="1.511013045s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:10:57.570334599 +0000 UTC m=+68.766850336" watchObservedRunningTime="2026-09-10 12:10:57.571223505 +0000 UTC m=+68.767739205"
Sep 10 12:10:58.520200 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:10:58.520170    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-xxvfc"
Sep 10 12:11:00.564767 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.564675    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w7g5k" event={"ID":"c982e9cd-3e88-4e3e-9d59-4bbca5053412","Type":"ContainerStarted","Data":"698af857f663a04ea301c596ef6bef930be1d91678badc37a8df167546707592"}
Sep 10 12:11:00.566187 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.566159    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-hdnv2" event={"ID":"ee0458f0-6820-424d-b378-1424b0bb7e24","Type":"ContainerStarted","Data":"d7f64f45da6a80339bff4a4d0ec7699ed654f95e245c978037bf9b2afce7342a"}
Sep 10 12:11:00.566348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.566289    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:11:00.585887 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.585841    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w7g5k" podStartSLOduration=65.81906403 podStartE2EDuration="1m10.585827137s" totalImagesPullingTime="4.766763107s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:09:50 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:00.585291654 +0000 UTC m=+71.781807354" watchObservedRunningTime="2026-09-10 12:11:00.585827137 +0000 UTC m=+71.782342836"
Sep 10 12:11:00.603118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.603066    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-hdnv2" podStartSLOduration=67.132661934 podStartE2EDuration="1m11.603050905s" totalImagesPullingTime="4.470388971s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:09:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:00.602166287 +0000 UTC m=+71.798681986" watchObservedRunningTime="2026-09-10 12:11:00.603050905 +0000 UTC m=+71.799566605"
Sep 10 12:11:00.622768 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.622733    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"]
Sep 10 12:11:00.625647 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.625626    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.628360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.628319    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 12:11:00.628360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.628346    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 10 12:11:00.628360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.628356    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 10 12:11:00.628575 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.628337    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 12:11:00.628823 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.628796    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 12:11:00.628904 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.628889    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-c9d8q\""
Sep 10 12:11:00.645324 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.645302    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"]
Sep 10 12:11:00.648943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.648917    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-z8l75"]
Sep 10 12:11:00.651851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.651837    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.654022 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.654004    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 12:11:00.654388 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.654368    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 12:11:00.654505 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.654462    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-6pwvt\""
Sep 10 12:11:00.654796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.654777    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 12:11:00.681772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681751    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htgm\" (UniqueName: \"kubernetes.io/projected/adf510c5-b4ea-49ff-8724-edf51e127bef-kube-api-access-9htgm\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.681908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681784    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf510c5-b4ea-49ff-8724-edf51e127bef-metrics-client-ca\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.681908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681811    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l9vm7\" (UniqueName: \"kubernetes.io/projected/2e08cbc4-613b-4638-85be-84fbd1e335b8-kube-api-access-l9vm7\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.681908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681840    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-sys\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.681908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681867    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.682081 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681908    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e08cbc4-613b-4638-85be-84fbd1e335b8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.682081 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681952    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-wtmp\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.682081 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681969    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e08cbc4-613b-4638-85be-84fbd1e335b8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.682081 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.681991    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-tls\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.682081 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.682007    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-root\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.682081 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.682044    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e08cbc4-613b-4638-85be-84fbd1e335b8-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.682301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.682103    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-textfile\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.682301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.682131    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783164 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783130    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-tls\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783164 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783167    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-root\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783185    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e08cbc4-613b-4638-85be-84fbd1e335b8-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783204    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-textfile\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783220    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783280    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-root\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:11:00.783321    2552 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783344    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htgm\" (UniqueName: \"kubernetes.io/projected/adf510c5-b4ea-49ff-8724-edf51e127bef-kube-api-access-9htgm\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:11:00.783385    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-tls podName:adf510c5-b4ea-49ff-8724-edf51e127bef nodeName:}" failed. No retries permitted until 2026-09-10 12:11:01.283366826 +0000 UTC m=+72.479882520 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-tls") pod "node-exporter-z8l75" (UID: "adf510c5-b4ea-49ff-8724-edf51e127bef") : secret "node-exporter-tls" not found
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783406    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf510c5-b4ea-49ff-8724-edf51e127bef-metrics-client-ca\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783465    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l9vm7\" (UniqueName: \"kubernetes.io/projected/2e08cbc4-613b-4638-85be-84fbd1e335b8-kube-api-access-l9vm7\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.783907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783497    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-sys\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783529    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783573    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e08cbc4-613b-4638-85be-84fbd1e335b8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.783907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783606    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-wtmp\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783630    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e08cbc4-613b-4638-85be-84fbd1e335b8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.783907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783636    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-sys\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.783907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.783857    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-wtmp\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.784200 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.784063    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/2e08cbc4-613b-4638-85be-84fbd1e335b8-metrics-client-ca\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.784200 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.784147    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-accelerators-collector-config\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.784200 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.784178    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-textfile\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.784338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.784215    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/adf510c5-b4ea-49ff-8724-edf51e127bef-metrics-client-ca\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.785840 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.785819    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.786095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.786079    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/2e08cbc4-613b-4638-85be-84fbd1e335b8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.786482 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.786461    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/2e08cbc4-613b-4638-85be-84fbd1e335b8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.794598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.794576    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htgm\" (UniqueName: \"kubernetes.io/projected/adf510c5-b4ea-49ff-8724-edf51e127bef-kube-api-access-9htgm\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:00.794907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.794884    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l9vm7\" (UniqueName: \"kubernetes.io/projected/2e08cbc4-613b-4638-85be-84fbd1e335b8-kube-api-access-l9vm7\") pod \"openshift-state-metrics-657fd6c54-v8wsv\" (UID: \"2e08cbc4-613b-4638-85be-84fbd1e335b8\") " pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:00.934916 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:00.934888    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"
Sep 10 12:11:01.209913 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.209741    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv"]
Sep 10 12:11:01.214235 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:11:01.214200    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2e08cbc4_613b_4638_85be_84fbd1e335b8.slice/crio-c8a258522a413f7f823ab12b40bd8183d87ef46d773724ff1d86e4b55101d6a1 WatchSource:0}: Error finding container c8a258522a413f7f823ab12b40bd8183d87ef46d773724ff1d86e4b55101d6a1: Status 404 returned error can't find the container with id c8a258522a413f7f823ab12b40bd8183d87ef46d773724ff1d86e4b55101d6a1
Sep 10 12:11:01.288003 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.287954    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-tls\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:01.290486 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.290461    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/adf510c5-b4ea-49ff-8724-edf51e127bef-node-exporter-tls\") pod \"node-exporter-z8l75\" (UID: \"adf510c5-b4ea-49ff-8724-edf51e127bef\") " pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:01.560768 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.560676    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-z8l75"
Sep 10 12:11:01.572918 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:11:01.572888    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podadf510c5_b4ea_49ff_8724_edf51e127bef.slice/crio-20efee9d2cadc8bce42367ede5355d7a577a016664dc92e90ff2257cf6425653 WatchSource:0}: Error finding container 20efee9d2cadc8bce42367ede5355d7a577a016664dc92e90ff2257cf6425653: Status 404 returned error can't find the container with id 20efee9d2cadc8bce42367ede5355d7a577a016664dc92e90ff2257cf6425653
Sep 10 12:11:01.574656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.574625    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv" event={"ID":"2e08cbc4-613b-4638-85be-84fbd1e335b8","Type":"ContainerStarted","Data":"f9244582a8bec26f14e317c4fd46084db1c4de245f740525707e2a06f9c73826"}
Sep 10 12:11:01.574735 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.574698    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv" event={"ID":"2e08cbc4-613b-4638-85be-84fbd1e335b8","Type":"ContainerStarted","Data":"f60dca5a3a4d8559ce19c54e5a425fdf85c1702f83c2af462020c9ddd8e16b00"}
Sep 10 12:11:01.574735 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.574723    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv" event={"ID":"2e08cbc4-613b-4638-85be-84fbd1e335b8","Type":"ContainerStarted","Data":"c8a258522a413f7f823ab12b40bd8183d87ef46d773724ff1d86e4b55101d6a1"}
Sep 10 12:11:01.686348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.686318    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:11:01.690871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.690854    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.693476 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693457    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 10 12:11:01.693610 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693579    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 10 12:11:01.693678 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693622    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 10 12:11:01.693678 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693585    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 10 12:11:01.693772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693588    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 10 12:11:01.693772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693622    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 10 12:11:01.693772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693569    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 10 12:11:01.693772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.693750    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 10 12:11:01.694014 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.694000    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 10 12:11:01.694070 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.694013    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-5z2tz\""
Sep 10 12:11:01.708387 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.708366    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:11:01.792407 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792372    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792407 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792409    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltkth\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-kube-api-access-ltkth\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792450    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-out\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792469    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792484    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-web-config\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792497    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792514    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792534    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792586    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792636    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792667    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792688    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.792862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.792707    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.893523 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893484    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-out\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.893523 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893533    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.893785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893555    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-web-config\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.893785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893578    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.893785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893645    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.893785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893669    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.893785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893742    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.894079 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:11:01.893801    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle podName:35cfa595-7ed8-4294-ba8d-a393f2228a8f nodeName:}" failed. No retries permitted until 2026-09-10 12:11:02.393779951 +0000 UTC m=+73.590295630 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f") : configmap references non-existent config key: ca-bundle.crt
Sep 10 12:11:01.894079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893835    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.894079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893874    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.894079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893905    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.894079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893937    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.894079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.893967    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.894079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.894019    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltkth\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-kube-api-access-ltkth\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.894676 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.894137    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.895189 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.895165    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.896646 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.896624    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-out\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.896730 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.896693    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.896787 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:11:01.896737    2552 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 10 12:11:01.896857 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:11:01.896790    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls podName:35cfa595-7ed8-4294-ba8d-a393f2228a8f nodeName:}" failed. No retries permitted until 2026-09-10 12:11:02.396773592 +0000 UTC m=+73.593289283 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f") : secret "alertmanager-main-tls" not found
Sep 10 12:11:01.896975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.896914    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.896975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.896913    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-web-config\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.897543 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.897521    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.899003 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.898980    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-tls-assets\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.899090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.899021    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-volume\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.899222 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.899203    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:01.909496 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:01.909459    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltkth\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-kube-api-access-ltkth\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:02.399197 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:02.399166    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:02.399373 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:02.399246    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:02.399469 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:11:02.399375    2552 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 10 12:11:02.399469 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:11:02.399463    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls podName:35cfa595-7ed8-4294-ba8d-a393f2228a8f nodeName:}" failed. No retries permitted until 2026-09-10 12:11:03.399443164 +0000 UTC m=+74.595958860 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f") : secret "alertmanager-main-tls" not found
Sep 10 12:11:02.400261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:02.400241    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:02.581306 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:02.581211    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv" event={"ID":"2e08cbc4-613b-4638-85be-84fbd1e335b8","Type":"ContainerStarted","Data":"6f228b09120e6b789566843a58b26510bf9e90cdbd1538c380658fceeb5e381e"}
Sep 10 12:11:02.582654 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:02.582626    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8l75" event={"ID":"adf510c5-b4ea-49ff-8724-edf51e127bef","Type":"ContainerStarted","Data":"20efee9d2cadc8bce42367ede5355d7a577a016664dc92e90ff2257cf6425653"}
Sep 10 12:11:02.605577 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:02.605516    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-657fd6c54-v8wsv" podStartSLOduration=1.6405776589999999 podStartE2EDuration="2.605502707s" totalImagesPullingTime="964.925048ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:02.605184102 +0000 UTC m=+73.801699811" watchObservedRunningTime="2026-09-10 12:11:02.605502707 +0000 UTC m=+73.802018406"
Sep 10 12:11:03.405837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:03.405798    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:03.408219 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:03.408194    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:03.500354 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:03.500328    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:11:03.590468 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:03.590411    2552 generic.go:356] "Generic (PLEG): container finished" podID="adf510c5-b4ea-49ff-8724-edf51e127bef" containerID="f955cc7d2ff8cac0694cdb783c682a8a9ccb94c831f4ca98aa7f6c08f96d762b" exitCode=0
Sep 10 12:11:03.590786 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:03.590467    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8l75" event={"ID":"adf510c5-b4ea-49ff-8724-edf51e127bef","Type":"ContainerDied","Data":"f955cc7d2ff8cac0694cdb783c682a8a9ccb94c831f4ca98aa7f6c08f96d762b"}
Sep 10 12:11:03.784166 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:03.784074    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:11:03.785992 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:11:03.785967    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod35cfa595_7ed8_4294_ba8d_a393f2228a8f.slice/crio-817f1b5a92c84a917295006e961cb0c4793888bbc1bdfab70cad64ccdf4de7bc WatchSource:0}: Error finding container 817f1b5a92c84a917295006e961cb0c4793888bbc1bdfab70cad64ccdf4de7bc: Status 404 returned error can't find the container with id 817f1b5a92c84a917295006e961cb0c4793888bbc1bdfab70cad64ccdf4de7bc
Sep 10 12:11:04.595397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:04.595364    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerStarted","Data":"817f1b5a92c84a917295006e961cb0c4793888bbc1bdfab70cad64ccdf4de7bc"}
Sep 10 12:11:04.597488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:04.597458    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8l75" event={"ID":"adf510c5-b4ea-49ff-8724-edf51e127bef","Type":"ContainerStarted","Data":"1f2bd90613a0113ef614f05fd72aa38e0414a2fa78900662ca69c644daff6d3c"}
Sep 10 12:11:04.597603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:04.597508    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-z8l75" event={"ID":"adf510c5-b4ea-49ff-8724-edf51e127bef","Type":"ContainerStarted","Data":"7066a0a0e20ed20c4b49637b8ebd227ebbea7925b03552527e4d5251fc022231"}
Sep 10 12:11:04.619975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:04.619929    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-z8l75" podStartSLOduration=3.4630072370000002 podStartE2EDuration="4.619914582s" totalImagesPullingTime="1.088657179s" totalInitContainerRuntime="68.250166ms" isStatefulPod=true podCreationTimestamp="2026-09-10 12:11:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:04.619445863 +0000 UTC m=+75.815961572" watchObservedRunningTime="2026-09-10 12:11:04.619914582 +0000 UTC m=+75.816430281"
Sep 10 12:11:05.503525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.503500    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-dffdcb8ff-rfpg2"
Sep 10 12:11:05.602553 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.602517    2552 generic.go:356] "Generic (PLEG): container finished" podID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerID="714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4" exitCode=0
Sep 10 12:11:05.602982 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.602604    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4"}
Sep 10 12:11:05.881581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.881554    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-8f75f756c-q9g5f"]
Sep 10 12:11:05.884781 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.884766    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.890026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.890003    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Sep 10 12:11:05.890026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.890014    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-fz6xs\""
Sep 10 12:11:05.890026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.890024    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Sep 10 12:11:05.890244 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.890166    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Sep 10 12:11:05.890319 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.890300    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Sep 10 12:11:05.891249 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.891234    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Sep 10 12:11:05.897104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.897082    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-8i12ta5c71j38\""
Sep 10 12:11:05.904507 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.904489    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8f75f756c-q9g5f"]
Sep 10 12:11:05.925012 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.924988    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-serving-certs-ca-bundle\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.925103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.925015    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-federate-client-tls\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.925103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.925033    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-metrics-client-ca\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.925103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.925047    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4x4gk\" (UniqueName: \"kubernetes.io/projected/178e1b02-25a5-4199-bb03-40324606309c-kube-api-access-4x4gk\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.925103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.925068    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-telemeter-client-tls\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.925244 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.925114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-secret-telemeter-client\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.925244 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.925162    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:05.925244 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:05.925193    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026286 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026263    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026303    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-serving-certs-ca-bundle\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026319    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-federate-client-tls\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026336    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-metrics-client-ca\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026351    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4x4gk\" (UniqueName: \"kubernetes.io/projected/178e1b02-25a5-4199-bb03-40324606309c-kube-api-access-4x4gk\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026375    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-telemeter-client-tls\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026675 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026498    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-secret-telemeter-client\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.026675 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.026581    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.027095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.027068    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-serving-certs-ca-bundle\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.027357 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.027302    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-metrics-client-ca\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.027357 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.027349    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/178e1b02-25a5-4199-bb03-40324606309c-telemeter-trusted-ca-bundle\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.028967 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.028950    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-federate-client-tls\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.029116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.029088    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.029561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.029541    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-telemeter-client-tls\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.029605 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.029541    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/178e1b02-25a5-4199-bb03-40324606309c-secret-telemeter-client\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.034394 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.034370    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4x4gk\" (UniqueName: \"kubernetes.io/projected/178e1b02-25a5-4199-bb03-40324606309c-kube-api-access-4x4gk\") pod \"telemeter-client-8f75f756c-q9g5f\" (UID: \"178e1b02-25a5-4199-bb03-40324606309c\") " pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.193738 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.193693    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f"
Sep 10 12:11:06.471220 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.471186    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-8f75f756c-q9g5f"]
Sep 10 12:11:06.472484 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:11:06.472451    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod178e1b02_25a5_4199_bb03_40324606309c.slice/crio-a2a321a3d4537fd314e1da3880a833c0464e5a4f19ce3328016532c168b5fe33 WatchSource:0}: Error finding container a2a321a3d4537fd314e1da3880a833c0464e5a4f19ce3328016532c168b5fe33: Status 404 returned error can't find the container with id a2a321a3d4537fd314e1da3880a833c0464e5a4f19ce3328016532c168b5fe33
Sep 10 12:11:06.608623 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.608594    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f" event={"ID":"178e1b02-25a5-4199-bb03-40324606309c","Type":"ContainerStarted","Data":"a2a321a3d4537fd314e1da3880a833c0464e5a4f19ce3328016532c168b5fe33"}
Sep 10 12:11:06.892201 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.892171    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:11:06.896182 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.896158    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.899415 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.899393    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 12:11:06.899724 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.899702    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 12:11:06.899834 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.899791    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2754t0b8lut0t\""
Sep 10 12:11:06.899834 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.899790    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 12:11:06.900216 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900046    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 12:11:06.900216 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900059    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 12:11:06.900216 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900151    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lm2ss\""
Sep 10 12:11:06.900454 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900266    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 12:11:06.900569 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900551    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 12:11:06.900672 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900581    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 12:11:06.900672 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900641    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 12:11:06.900775 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900704    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 12:11:06.901023 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900883    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 12:11:06.901023 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.900940    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 12:11:06.901500 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.901480    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 12:11:06.901588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.901525    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 12:11:06.905765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.905742    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 12:11:06.915781 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.915761    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:11:06.935756 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935730    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935846 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935766    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vczv2\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-kube-api-access-vczv2\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935846 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935798    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935846 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935826    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935850    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935878    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935900    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935923    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935945    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935963    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.935995    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936067    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936121    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936140    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936162    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936206    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936230    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936249    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:06.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:06.936296    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037408    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037480    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037504    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037524    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037556    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037585    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037611    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037640    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037676    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037715    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037754    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037777    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037808    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037842    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037874    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037897    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vczv2\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-kube-api-access-vczv2\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037927    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037957    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.037979    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.038007    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.038851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.038747    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.039378 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.039348    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.040862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.040312    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.040862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.040468    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.040862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.040584    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.040862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.040822    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.041108 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.040870    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.041664 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.041640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.042831 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.042807    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.043609 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.043567    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.044234 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.044207    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.044320 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.044243    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.044980 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.044956    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.045104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.045074    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.045216 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.045150    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.045648 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.045625    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config-out\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.045803 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.045778    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.045958 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.045936    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-web-config\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.046176 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.046155    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.048898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.048875    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vczv2\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-kube-api-access-vczv2\") pod \"prometheus-k8s-0\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.209776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.209750    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:07.509615 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.509584    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:11:07.510579 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:11:07.510554    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3beb8dc2_7eb0_4f09_98fe_410ebd0097fb.slice/crio-26f3a564f0b4eb3a9bcc5d4750729312334c309132bd985b9f2cd6379c6c2ca6 WatchSource:0}: Error finding container 26f3a564f0b4eb3a9bcc5d4750729312334c309132bd985b9f2cd6379c6c2ca6: Status 404 returned error can't find the container with id 26f3a564f0b4eb3a9bcc5d4750729312334c309132bd985b9f2cd6379c6c2ca6
Sep 10 12:11:07.615995 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.615966    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerStarted","Data":"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"}
Sep 10 12:11:07.616338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.616039    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerStarted","Data":"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"}
Sep 10 12:11:07.616338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.616073    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerStarted","Data":"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"}
Sep 10 12:11:07.616338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.616109    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerStarted","Data":"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"}
Sep 10 12:11:07.616338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.616147    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerStarted","Data":"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"}
Sep 10 12:11:07.617643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.617622    2552 generic.go:356] "Generic (PLEG): container finished" podID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a" exitCode=0
Sep 10 12:11:07.617763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.617653    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"}
Sep 10 12:11:07.617763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:07.617718    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerStarted","Data":"26f3a564f0b4eb3a9bcc5d4750729312334c309132bd985b9f2cd6379c6c2ca6"}
Sep 10 12:11:09.634607 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:09.634561    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerStarted","Data":"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"}
Sep 10 12:11:09.637103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:09.637072    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f" event={"ID":"178e1b02-25a5-4199-bb03-40324606309c","Type":"ContainerStarted","Data":"9bcc816ce69c6dd2000aea269d8fb28eb0d14a3f216c1e21fcf522b8d8dc70e4"}
Sep 10 12:11:09.637243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:09.637127    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f" event={"ID":"178e1b02-25a5-4199-bb03-40324606309c","Type":"ContainerStarted","Data":"4af6a9ea8ed3a109f4cf44611676f9be077fbecd42004d5a9662cd42650a1688"}
Sep 10 12:11:09.637243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:09.637158    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f" event={"ID":"178e1b02-25a5-4199-bb03-40324606309c","Type":"ContainerStarted","Data":"16488e068b8ab0f03d915cb2271095108c74687eac0ba6dcbb7cae05c48e0bbc"}
Sep 10 12:11:09.664788 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:09.664586    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=4.482954858 podStartE2EDuration="8.664560178s" totalImagesPullingTime="4.166340245s" totalInitContainerRuntime="15.265075ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:11:01 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:09.661258194 +0000 UTC m=+80.857773941" watchObservedRunningTime="2026-09-10 12:11:09.664560178 +0000 UTC m=+80.861075880"
Sep 10 12:11:09.686537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:09.686478    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-8f75f756c-q9g5f" podStartSLOduration=2.570849858 podStartE2EDuration="4.686452943s" totalImagesPullingTime="2.115603085s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:11:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:09.684753215 +0000 UTC m=+80.881268916" watchObservedRunningTime="2026-09-10 12:11:09.686452943 +0000 UTC m=+80.882968645"
Sep 10 12:11:11.648246 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:11.648205    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerStarted","Data":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"}
Sep 10 12:11:11.648663 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:11.648297    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerStarted","Data":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"}
Sep 10 12:11:13.663357 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:13.663319    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerStarted","Data":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"}
Sep 10 12:11:13.663850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:13.663441    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerStarted","Data":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"}
Sep 10 12:11:13.663850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:13.663494    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerStarted","Data":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"}
Sep 10 12:11:13.663850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:13.663537    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerStarted","Data":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"}
Sep 10 12:11:13.702682 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:13.698892    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.495342101 podStartE2EDuration="7.69886963s" totalImagesPullingTime="5.184071935s" totalInitContainerRuntime="19.455594ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:11:06 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:11:13.6966458 +0000 UTC m=+84.893161502" watchObservedRunningTime="2026-09-10 12:11:13.69886963 +0000 UTC m=+84.895385332"
Sep 10 12:11:17.210530 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:17.210497    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:11:31.577332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:11:31.577299    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-hdnv2"
Sep 10 12:12:07.210515 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:07.210471    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:07.230290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:07.230264    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:07.870911 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:07.870882    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:21.024811 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.024777    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:12:21.025214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.025037    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="alertmanager" containerID="cri-o://7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51" gracePeriod=120
Sep 10 12:12:21.025214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.025114    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-web" containerID="cri-o://7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d" gracePeriod=120
Sep 10 12:12:21.025214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.025114    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="prom-label-proxy" containerID="cri-o://ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04" gracePeriod=120
Sep 10 12:12:21.025214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.025142    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy" containerID="cri-o://3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040" gracePeriod=120
Sep 10 12:12:21.025214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.025209    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="config-reloader" containerID="cri-o://63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7" gracePeriod=120
Sep 10 12:12:21.025507 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.025285    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-metric" containerID="cri-o://fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388" gracePeriod=120
Sep 10 12:12:21.907255 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907218    2552 generic.go:356] "Generic (PLEG): container finished" podID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerID="ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04" exitCode=0
Sep 10 12:12:21.907255 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907251    2552 generic.go:356] "Generic (PLEG): container finished" podID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerID="3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040" exitCode=0
Sep 10 12:12:21.907448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907261    2552 generic.go:356] "Generic (PLEG): container finished" podID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerID="63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7" exitCode=0
Sep 10 12:12:21.907448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907260    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"}
Sep 10 12:12:21.907448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907326    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"}
Sep 10 12:12:21.907448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907270    2552 generic.go:356] "Generic (PLEG): container finished" podID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerID="7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51" exitCode=0
Sep 10 12:12:21.907448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907369    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"}
Sep 10 12:12:21.907448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:21.907409    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"}
Sep 10 12:12:22.265904 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.265882    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:12:22.334560 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334525    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-web-config\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334577    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-web\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334612    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-metric\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334646    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-cluster-tls-config\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334689    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334723    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ltkth\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-kube-api-access-ltkth\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334748    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-out\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334807    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-tls-assets\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334833    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-main-db\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334856    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-volume\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334890    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.334928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334920    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-metrics-client-ca\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.335259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.334949    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle\") pod \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\" (UID: \"35cfa595-7ed8-4294-ba8d-a393f2228a8f\") "
Sep 10 12:12:22.335514 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.335488    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle" (OuterVolumeSpecName: "alertmanager-trusted-ca-bundle") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:22.335674 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.335482    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-main-db" (OuterVolumeSpecName: "alertmanager-main-db") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:12:22.337742 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.337708    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-web") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:22.338218 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.337968    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls" (OuterVolumeSpecName: "secret-alertmanager-main-tls") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:22.338318 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.338279    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:12:22.338386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.338316    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-metric" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy-metric") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:22.338465 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.338386    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-out" (OuterVolumeSpecName: "config-out") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:12:22.338737 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.338701    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-metrics-client-ca" (OuterVolumeSpecName: "metrics-client-ca") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:22.338985 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.338962    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-volume" (OuterVolumeSpecName: "config-volume") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:22.340196 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.340173    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-kube-api-access-ltkth" (OuterVolumeSpecName: "kube-api-access-ltkth") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "kube-api-access-ltkth". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:12:22.340597 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.340577    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy" (OuterVolumeSpecName: "secret-alertmanager-kube-rbac-proxy") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:22.343001 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.342974    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-cluster-tls-config" (OuterVolumeSpecName: "cluster-tls-config") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:22.348377 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.348355    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-web-config" (OuterVolumeSpecName: "web-config") pod "35cfa595-7ed8-4294-ba8d-a393f2228a8f" (UID: "35cfa595-7ed8-4294-ba8d-a393f2228a8f"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:22.435697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435671    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-web-config\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435694    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435710    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435724    2552 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-cluster-tls-config\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435737    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-main-tls\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435750    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltkth\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-kube-api-access-ltkth\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435762    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-out\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435774    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/35cfa595-7ed8-4294-ba8d-a393f2228a8f-tls-assets\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435786    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-main-db\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435798    2552 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435811    2552 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/35cfa595-7ed8-4294-ba8d-a393f2228a8f-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435824    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-metrics-client-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.435851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.435837    2552 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/35cfa595-7ed8-4294-ba8d-a393f2228a8f-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:22.916464 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.916413    2552 generic.go:356] "Generic (PLEG): container finished" podID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerID="fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388" exitCode=0
Sep 10 12:12:22.916464 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.916465    2552 generic.go:356] "Generic (PLEG): container finished" podID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerID="7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d" exitCode=0
Sep 10 12:12:22.916639 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.916473    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"}
Sep 10 12:12:22.916639 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.916558    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 10 12:12:22.916639 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.916596    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"}
Sep 10 12:12:22.916639 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.916629    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"35cfa595-7ed8-4294-ba8d-a393f2228a8f","Type":"ContainerDied","Data":"817f1b5a92c84a917295006e961cb0c4793888bbc1bdfab70cad64ccdf4de7bc"}
Sep 10 12:12:22.916793 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.916675    2552 scope.go:117] "RemoveContainer" containerID="ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"
Sep 10 12:12:22.924668 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.924648    2552 scope.go:117] "RemoveContainer" containerID="fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"
Sep 10 12:12:22.931771 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.931756    2552 scope.go:117] "RemoveContainer" containerID="3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"
Sep 10 12:12:22.938806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.938780    2552 scope.go:117] "RemoveContainer" containerID="7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"
Sep 10 12:12:22.939885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.939861    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:12:22.943765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.943743    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 10 12:12:22.946121 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.946102    2552 scope.go:117] "RemoveContainer" containerID="63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"
Sep 10 12:12:22.952805 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.952788    2552 scope.go:117] "RemoveContainer" containerID="7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"
Sep 10 12:12:22.959552 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.959514    2552 scope.go:117] "RemoveContainer" containerID="714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4"
Sep 10 12:12:22.966032 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.966016    2552 scope.go:117] "RemoveContainer" containerID="ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"
Sep 10 12:12:22.966299 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:22.966279    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04\": container with ID starting with ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04 not found: ID does not exist" containerID="ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"
Sep 10 12:12:22.966368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.966311    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"} err="failed to get container status \"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04\": rpc error: code = NotFound desc = could not find container \"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04\": container with ID starting with ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04 not found: ID does not exist"
Sep 10 12:12:22.966368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.966353    2552 scope.go:117] "RemoveContainer" containerID="fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"
Sep 10 12:12:22.966635 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:22.966618    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388\": container with ID starting with fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388 not found: ID does not exist" containerID="fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"
Sep 10 12:12:22.966675 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.966650    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"} err="failed to get container status \"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388\": rpc error: code = NotFound desc = could not find container \"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388\": container with ID starting with fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388 not found: ID does not exist"
Sep 10 12:12:22.966675 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.966665    2552 scope.go:117] "RemoveContainer" containerID="3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"
Sep 10 12:12:22.966913 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:22.966891    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040\": container with ID starting with 3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040 not found: ID does not exist" containerID="3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"
Sep 10 12:12:22.966950 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.966919    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"} err="failed to get container status \"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040\": rpc error: code = NotFound desc = could not find container \"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040\": container with ID starting with 3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040 not found: ID does not exist"
Sep 10 12:12:22.966950 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.966933    2552 scope.go:117] "RemoveContainer" containerID="7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"
Sep 10 12:12:22.967176 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:22.967157    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d\": container with ID starting with 7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d not found: ID does not exist" containerID="7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"
Sep 10 12:12:22.967236 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967184    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"} err="failed to get container status \"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d\": rpc error: code = NotFound desc = could not find container \"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d\": container with ID starting with 7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d not found: ID does not exist"
Sep 10 12:12:22.967236 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967205    2552 scope.go:117] "RemoveContainer" containerID="63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"
Sep 10 12:12:22.967454 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:22.967420    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7\": container with ID starting with 63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7 not found: ID does not exist" containerID="63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"
Sep 10 12:12:22.967501 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967460    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"} err="failed to get container status \"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7\": rpc error: code = NotFound desc = could not find container \"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7\": container with ID starting with 63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7 not found: ID does not exist"
Sep 10 12:12:22.967501 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967474    2552 scope.go:117] "RemoveContainer" containerID="7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"
Sep 10 12:12:22.967690 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:22.967676    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51\": container with ID starting with 7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51 not found: ID does not exist" containerID="7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"
Sep 10 12:12:22.967731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967692    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"} err="failed to get container status \"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51\": rpc error: code = NotFound desc = could not find container \"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51\": container with ID starting with 7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51 not found: ID does not exist"
Sep 10 12:12:22.967731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967704    2552 scope.go:117] "RemoveContainer" containerID="714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4"
Sep 10 12:12:22.967927 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:22.967909    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4\": container with ID starting with 714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4 not found: ID does not exist" containerID="714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4"
Sep 10 12:12:22.968158 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967933    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4"} err="failed to get container status \"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4\": rpc error: code = NotFound desc = could not find container \"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4\": container with ID starting with 714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4 not found: ID does not exist"
Sep 10 12:12:22.968158 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.967951    2552 scope.go:117] "RemoveContainer" containerID="ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"
Sep 10 12:12:22.968381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.968208    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04"} err="failed to get container status \"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04\": rpc error: code = NotFound desc = could not find container \"ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04\": container with ID starting with ff4a71bb2b295bd0edb6a57830441f10055f6c1b556a87a1bbf58716436aab04 not found: ID does not exist"
Sep 10 12:12:22.968381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.968230    2552 scope.go:117] "RemoveContainer" containerID="fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"
Sep 10 12:12:22.968676 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.968648    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388"} err="failed to get container status \"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388\": rpc error: code = NotFound desc = could not find container \"fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388\": container with ID starting with fea007a19e6903f24f04feedaff1741a121991075d4a6085ec06d2bf1a683388 not found: ID does not exist"
Sep 10 12:12:22.968787 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.968677    2552 scope.go:117] "RemoveContainer" containerID="3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"
Sep 10 12:12:22.968974 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.968953    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040"} err="failed to get container status \"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040\": rpc error: code = NotFound desc = could not find container \"3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040\": container with ID starting with 3ab6080d9aff049250734f7f84a29402f9ce07add121902e17377eb4cd1ee040 not found: ID does not exist"
Sep 10 12:12:22.969050 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.968975    2552 scope.go:117] "RemoveContainer" containerID="7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"
Sep 10 12:12:22.969242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.969212    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d"} err="failed to get container status \"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d\": rpc error: code = NotFound desc = could not find container \"7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d\": container with ID starting with 7c407b4df29403cde6279090a370f10bf41574c43fb89338ec00c8ac9d42e15d not found: ID does not exist"
Sep 10 12:12:22.969242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.969235    2552 scope.go:117] "RemoveContainer" containerID="63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"
Sep 10 12:12:22.969482 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.969466    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7"} err="failed to get container status \"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7\": rpc error: code = NotFound desc = could not find container \"63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7\": container with ID starting with 63d0e20c2384ee02b63aec7770ac1858bb94385b914ee51636bf8c0279e7dba7 not found: ID does not exist"
Sep 10 12:12:22.969538 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.969483    2552 scope.go:117] "RemoveContainer" containerID="7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"
Sep 10 12:12:22.969701 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.969683    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51"} err="failed to get container status \"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51\": rpc error: code = NotFound desc = could not find container \"7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51\": container with ID starting with 7e6ab63fb7ace8bed756dbc19ac9aa4074f4f9776ef27c3cd3d5cfb912db9c51 not found: ID does not exist"
Sep 10 12:12:22.969753 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.969701    2552 scope.go:117] "RemoveContainer" containerID="714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4"
Sep 10 12:12:22.969926 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:22.969910    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4"} err="failed to get container status \"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4\": rpc error: code = NotFound desc = could not find container \"714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4\": container with ID starting with 714154e459e748239b64836dc5601aa94296f8e0b20f1b683fd503caba6d58e4 not found: ID does not exist"
Sep 10 12:12:23.240766 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:23.240705    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" path="/var/lib/kubelet/pods/35cfa595-7ed8-4294-ba8d-a393f2228a8f/volumes"
Sep 10 12:12:25.252142 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.252112    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:12:25.252656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.252391    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="prometheus" containerID="cri-o://af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209" gracePeriod=600
Sep 10 12:12:25.252656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.252410    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy" containerID="cri-o://bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21" gracePeriod=600
Sep 10 12:12:25.252656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.252448    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="thanos-sidecar" containerID="cri-o://dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9" gracePeriod=600
Sep 10 12:12:25.252656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.252498    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-web" containerID="cri-o://4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d" gracePeriod=600
Sep 10 12:12:25.252656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.252521    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-thanos" containerID="cri-o://9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba" gracePeriod=600
Sep 10 12:12:25.252656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.252526    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="config-reloader" containerID="cri-o://7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327" gracePeriod=600
Sep 10 12:12:25.485170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.485149    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:25.557834 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.557772    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vczv2\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-kube-api-access-vczv2\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.557834 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.557801    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config-out\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.557834 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.557822    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.557845    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-serving-certs-ca-bundle\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.557928    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-db\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.557967    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-tls-assets\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.557993    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-kube-rbac-proxy\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558035    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-tls\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558066    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-0\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558098    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-2\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558146    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-metrics-client-certs\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558177    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-1\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558201    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-metrics-client-ca\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558200    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-serving-certs-ca-bundle" (OuterVolumeSpecName: "configmap-serving-certs-ca-bundle") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558230    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-trusted-ca-bundle\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558272    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-kubelet-serving-ca-bundle\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558308    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-thanos-prometheus-http-client-file\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558339    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-web-config\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558365    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558857 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558401    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-grpc-tls\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558857 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558447    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config\") pod \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\" (UID: \"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb\") "
Sep 10 12:12:25.558857 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.558665    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.559772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.559703    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-2" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-2") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:25.559890 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.559832    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-kubelet-serving-ca-bundle" (OuterVolumeSpecName: "configmap-kubelet-serving-ca-bundle") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:25.561776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.560238    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-metrics-client-ca" (OuterVolumeSpecName: "configmap-metrics-client-ca") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:25.561776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.560644    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-0" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-0") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:25.562121 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.562064    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-1" (OuterVolumeSpecName: "prometheus-k8s-rulefiles-1") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:25.562237 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.562189    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-trusted-ca-bundle" (OuterVolumeSpecName: "prometheus-trusted-ca-bundle") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:12:25.562418 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.562395    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-db" (OuterVolumeSpecName: "prometheus-k8s-db") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:12:25.562642 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.562616    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-tls") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.562816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.562791    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-metrics-client-certs" (OuterVolumeSpecName: "secret-metrics-client-certs") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.562911 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.562887    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config-out" (OuterVolumeSpecName: "config-out") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:12:25.562965 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.562928    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-thanos-sidecar-tls" (OuterVolumeSpecName: "secret-prometheus-k8s-thanos-sidecar-tls") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.563153 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.563128    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-tls-assets" (OuterVolumeSpecName: "tls-assets") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:12:25.563245 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.563220    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-kube-api-access-vczv2" (OuterVolumeSpecName: "kube-api-access-vczv2") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "kube-api-access-vczv2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:12:25.563311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.563299    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-kube-rbac-proxy" (OuterVolumeSpecName: "secret-kube-rbac-proxy") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.563387 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.563368    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config" (OuterVolumeSpecName: "config") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.563897 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.563866    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-kube-rbac-proxy-web" (OuterVolumeSpecName: "secret-prometheus-k8s-kube-rbac-proxy-web") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.563990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.563940    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-grpc-tls" (OuterVolumeSpecName: "secret-grpc-tls") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.564132 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.564108    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-thanos-prometheus-http-client-file" (OuterVolumeSpecName: "thanos-prometheus-http-client-file") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.573524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.573499    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-web-config" (OuterVolumeSpecName: "web-config") pod "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" (UID: "3beb8dc2-7eb0-4f09-98fe-410ebd0097fb"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:12:25.659272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659228    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vczv2\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-kube-api-access-vczv2\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659273    2552 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config-out\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659284    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659294    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-db\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659302    2552 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-tls-assets\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659310    2552 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-kube-rbac-proxy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659318    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-tls\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659326    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659334    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659341    2552 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-metrics-client-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659349    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659358    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-metrics-client-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659366    2552 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-prometheus-trusted-ca-bundle\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659375    2552 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659383    2552 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-thanos-prometheus-http-client-file\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659391    2552 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-web-config\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659399    2552 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659407    2552 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-secret-grpc-tls\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.659798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.659415    2552 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb-config\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:12:25.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.935959    2552 generic.go:356] "Generic (PLEG): container finished" podID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba" exitCode=0
Sep 10 12:12:25.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.935981    2552 generic.go:356] "Generic (PLEG): container finished" podID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21" exitCode=0
Sep 10 12:12:25.935988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.935989    2552 generic.go:356] "Generic (PLEG): container finished" podID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d" exitCode=0
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.935995    2552 generic.go:356] "Generic (PLEG): container finished" podID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9" exitCode=0
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936000    2552 generic.go:356] "Generic (PLEG): container finished" podID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327" exitCode=0
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936005    2552 generic.go:356] "Generic (PLEG): container finished" podID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209" exitCode=0
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936035    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"}
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936046    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936122    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"}
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936149    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"}
Sep 10 12:12:25.936172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936174    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"}
Sep 10 12:12:25.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936188    2552 scope.go:117] "RemoveContainer" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936198    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"}
Sep 10 12:12:25.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936221    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"}
Sep 10 12:12:25.936452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.936243    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"3beb8dc2-7eb0-4f09-98fe-410ebd0097fb","Type":"ContainerDied","Data":"26f3a564f0b4eb3a9bcc5d4750729312334c309132bd985b9f2cd6379c6c2ca6"}
Sep 10 12:12:25.944690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.944672    2552 scope.go:117] "RemoveContainer" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.951253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.951239    2552 scope.go:117] "RemoveContainer" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.957749 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.957727    2552 scope.go:117] "RemoveContainer" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.959310 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.959290    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:12:25.964192 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.964174    2552 scope.go:117] "RemoveContainer" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.965906 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.965887    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:12:25.970836 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.970822    2552 scope.go:117] "RemoveContainer" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.977294 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.977277    2552 scope.go:117] "RemoveContainer" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.983483 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.983467    2552 scope.go:117] "RemoveContainer" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.983759 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:25.983741    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": container with ID starting with 9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba not found: ID does not exist" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.983816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.983766    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"} err="failed to get container status \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": rpc error: code = NotFound desc = could not find container \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": container with ID starting with 9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba not found: ID does not exist"
Sep 10 12:12:25.983816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.983784    2552 scope.go:117] "RemoveContainer" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.984031 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:25.984012    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": container with ID starting with bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21 not found: ID does not exist" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.984073 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984040    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"} err="failed to get container status \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": rpc error: code = NotFound desc = could not find container \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": container with ID starting with bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21 not found: ID does not exist"
Sep 10 12:12:25.984073 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984057    2552 scope.go:117] "RemoveContainer" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.984316 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:25.984300    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": container with ID starting with 4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d not found: ID does not exist" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.984371 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984323    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"} err="failed to get container status \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": rpc error: code = NotFound desc = could not find container \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": container with ID starting with 4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d not found: ID does not exist"
Sep 10 12:12:25.984371 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984343    2552 scope.go:117] "RemoveContainer" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.984640 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:25.984624    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": container with ID starting with dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9 not found: ID does not exist" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.984680 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984644    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"} err="failed to get container status \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": rpc error: code = NotFound desc = could not find container \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": container with ID starting with dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9 not found: ID does not exist"
Sep 10 12:12:25.984680 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984660    2552 scope.go:117] "RemoveContainer" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.984880 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:25.984866    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": container with ID starting with 7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327 not found: ID does not exist" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.984923 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984883    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"} err="failed to get container status \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": rpc error: code = NotFound desc = could not find container \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": container with ID starting with 7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327 not found: ID does not exist"
Sep 10 12:12:25.984923 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.984894    2552 scope.go:117] "RemoveContainer" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.985131 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:25.985114    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": container with ID starting with af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209 not found: ID does not exist" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.985177 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985136    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"} err="failed to get container status \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": rpc error: code = NotFound desc = could not find container \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": container with ID starting with af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209 not found: ID does not exist"
Sep 10 12:12:25.985177 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985150    2552 scope.go:117] "RemoveContainer" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.985373 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:12:25.985357    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": container with ID starting with 8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a not found: ID does not exist" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.985421 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985377    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"} err="failed to get container status \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": rpc error: code = NotFound desc = could not find container \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": container with ID starting with 8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a not found: ID does not exist"
Sep 10 12:12:25.985421 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985390    2552 scope.go:117] "RemoveContainer" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.985638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985619    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"} err="failed to get container status \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": rpc error: code = NotFound desc = could not find container \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": container with ID starting with 9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba not found: ID does not exist"
Sep 10 12:12:25.985676 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985640    2552 scope.go:117] "RemoveContainer" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.985855 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985838    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"} err="failed to get container status \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": rpc error: code = NotFound desc = could not find container \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": container with ID starting with bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21 not found: ID does not exist"
Sep 10 12:12:25.985894 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.985856    2552 scope.go:117] "RemoveContainer" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.986076 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986060    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"} err="failed to get container status \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": rpc error: code = NotFound desc = could not find container \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": container with ID starting with 4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d not found: ID does not exist"
Sep 10 12:12:25.986116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986076    2552 scope.go:117] "RemoveContainer" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.986310 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986286    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"} err="failed to get container status \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": rpc error: code = NotFound desc = could not find container \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": container with ID starting with dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9 not found: ID does not exist"
Sep 10 12:12:25.986310 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986307    2552 scope.go:117] "RemoveContainer" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.986554 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986537    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"} err="failed to get container status \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": rpc error: code = NotFound desc = could not find container \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": container with ID starting with 7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327 not found: ID does not exist"
Sep 10 12:12:25.986634 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986556    2552 scope.go:117] "RemoveContainer" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.986795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986776    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"} err="failed to get container status \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": rpc error: code = NotFound desc = could not find container \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": container with ID starting with af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209 not found: ID does not exist"
Sep 10 12:12:25.986845 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986795    2552 scope.go:117] "RemoveContainer" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.987006 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.986990    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"} err="failed to get container status \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": rpc error: code = NotFound desc = could not find container \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": container with ID starting with 8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a not found: ID does not exist"
Sep 10 12:12:25.987073 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987007    2552 scope.go:117] "RemoveContainer" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.987222 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987202    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"} err="failed to get container status \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": rpc error: code = NotFound desc = could not find container \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": container with ID starting with 9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba not found: ID does not exist"
Sep 10 12:12:25.987263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987223    2552 scope.go:117] "RemoveContainer" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.987397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987380    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"} err="failed to get container status \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": rpc error: code = NotFound desc = could not find container \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": container with ID starting with bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21 not found: ID does not exist"
Sep 10 12:12:25.987488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987398    2552 scope.go:117] "RemoveContainer" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.987626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987611    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"} err="failed to get container status \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": rpc error: code = NotFound desc = could not find container \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": container with ID starting with 4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d not found: ID does not exist"
Sep 10 12:12:25.987666 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987626    2552 scope.go:117] "RemoveContainer" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.987821 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987803    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"} err="failed to get container status \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": rpc error: code = NotFound desc = could not find container \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": container with ID starting with dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9 not found: ID does not exist"
Sep 10 12:12:25.987880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.987822    2552 scope.go:117] "RemoveContainer" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.988050 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988031    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"} err="failed to get container status \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": rpc error: code = NotFound desc = could not find container \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": container with ID starting with 7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327 not found: ID does not exist"
Sep 10 12:12:25.988089 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988051    2552 scope.go:117] "RemoveContainer" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.988243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988225    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"} err="failed to get container status \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": rpc error: code = NotFound desc = could not find container \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": container with ID starting with af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209 not found: ID does not exist"
Sep 10 12:12:25.988307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988244    2552 scope.go:117] "RemoveContainer" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.988492 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988474    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"} err="failed to get container status \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": rpc error: code = NotFound desc = could not find container \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": container with ID starting with 8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a not found: ID does not exist"
Sep 10 12:12:25.988551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988492    2552 scope.go:117] "RemoveContainer" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.988699 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988677    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"} err="failed to get container status \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": rpc error: code = NotFound desc = could not find container \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": container with ID starting with 9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba not found: ID does not exist"
Sep 10 12:12:25.988699 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988700    2552 scope.go:117] "RemoveContainer" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.988951 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988934    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"} err="failed to get container status \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": rpc error: code = NotFound desc = could not find container \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": container with ID starting with bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21 not found: ID does not exist"
Sep 10 12:12:25.989005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.988952    2552 scope.go:117] "RemoveContainer" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.989168 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989153    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"} err="failed to get container status \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": rpc error: code = NotFound desc = could not find container \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": container with ID starting with 4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d not found: ID does not exist"
Sep 10 12:12:25.989168 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989168    2552 scope.go:117] "RemoveContainer" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.989380 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989360    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"} err="failed to get container status \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": rpc error: code = NotFound desc = could not find container \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": container with ID starting with dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9 not found: ID does not exist"
Sep 10 12:12:25.989420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989381    2552 scope.go:117] "RemoveContainer" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.989608 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989592    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"} err="failed to get container status \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": rpc error: code = NotFound desc = could not find container \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": container with ID starting with 7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327 not found: ID does not exist"
Sep 10 12:12:25.989648 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989608    2552 scope.go:117] "RemoveContainer" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.989819 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989802    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"} err="failed to get container status \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": rpc error: code = NotFound desc = could not find container \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": container with ID starting with af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209 not found: ID does not exist"
Sep 10 12:12:25.989871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.989819    2552 scope.go:117] "RemoveContainer" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.990028 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990013    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"} err="failed to get container status \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": rpc error: code = NotFound desc = could not find container \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": container with ID starting with 8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a not found: ID does not exist"
Sep 10 12:12:25.990069 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990029    2552 scope.go:117] "RemoveContainer" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.990231 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990214    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"} err="failed to get container status \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": rpc error: code = NotFound desc = could not find container \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": container with ID starting with 9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba not found: ID does not exist"
Sep 10 12:12:25.990267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990231    2552 scope.go:117] "RemoveContainer" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.990466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990449    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"} err="failed to get container status \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": rpc error: code = NotFound desc = could not find container \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": container with ID starting with bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21 not found: ID does not exist"
Sep 10 12:12:25.990524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990467    2552 scope.go:117] "RemoveContainer" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.990669 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990653    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"} err="failed to get container status \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": rpc error: code = NotFound desc = could not find container \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": container with ID starting with 4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d not found: ID does not exist"
Sep 10 12:12:25.990717 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.990670    2552 scope.go:117] "RemoveContainer" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.991052 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.991027    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"} err="failed to get container status \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": rpc error: code = NotFound desc = could not find container \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": container with ID starting with dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9 not found: ID does not exist"
Sep 10 12:12:25.991158 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.991051    2552 scope.go:117] "RemoveContainer" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.991403 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.991380    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"} err="failed to get container status \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": rpc error: code = NotFound desc = could not find container \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": container with ID starting with 7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327 not found: ID does not exist"
Sep 10 12:12:25.991522 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.991404    2552 scope.go:117] "RemoveContainer" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.991837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.991815    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"} err="failed to get container status \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": rpc error: code = NotFound desc = could not find container \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": container with ID starting with af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209 not found: ID does not exist"
Sep 10 12:12:25.991837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.991838    2552 scope.go:117] "RemoveContainer" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.992119 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992099    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"} err="failed to get container status \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": rpc error: code = NotFound desc = could not find container \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": container with ID starting with 8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a not found: ID does not exist"
Sep 10 12:12:25.992191 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992120    2552 scope.go:117] "RemoveContainer" containerID="9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"
Sep 10 12:12:25.992358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992340    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:12:25.992416 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992362    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba"} err="failed to get container status \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": rpc error: code = NotFound desc = could not find container \"9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba\": container with ID starting with 9e7d7506e57e761544e57a7c06aefc3c32a3c526c6e77e281d38e43d6e48efba not found: ID does not exist"
Sep 10 12:12:25.992416 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992384    2552 scope.go:117] "RemoveContainer" containerID="bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"
Sep 10 12:12:25.992603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992590    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="thanos-sidecar"
Sep 10 12:12:25.992635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992604    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="thanos-sidecar"
Sep 10 12:12:25.992635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992618    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy"
Sep 10 12:12:25.992635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992626    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy"
Sep 10 12:12:25.992635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992623    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21"} err="failed to get container status \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": rpc error: code = NotFound desc = could not find container \"bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21\": container with ID starting with bb3162b2307ff8c1662cf8cec785c656ad184f1629cbae94c6418f4df1943a21 not found: ID does not exist"
Sep 10 12:12:25.992635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992633    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-metric"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992638    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-metric"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992639    2552 scope.go:117] "RemoveContainer" containerID="4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992646    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="prom-label-proxy"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992652    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="prom-label-proxy"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992658    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="alertmanager"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992663    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="alertmanager"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992670    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-web"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992675    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-web"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992680    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="init-config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992685    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="init-config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992692    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992697    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992702    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992706    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992712    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992716    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992724    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-web"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992732    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-web"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992741    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-thanos"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992749    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-thanos"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992757    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="prometheus"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992761    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="prometheus"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992768    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="init-config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992773    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="init-config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992822    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="config-reloader"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992829    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy"
Sep 10 12:12:25.992817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992835    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-metric"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992841    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="alertmanager"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992846    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="thanos-sidecar"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992851    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="kube-rbac-proxy-web"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992855    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992854    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d"} err="failed to get container status \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": rpc error: code = NotFound desc = could not find container \"4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d\": container with ID starting with 4301ee95457213fe8c458d321fa25bb34c8e3b9a89607a410e464dd567a5188d not found: ID does not exist"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992873    2552 scope.go:117] "RemoveContainer" containerID="dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992861    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="35cfa595-7ed8-4294-ba8d-a393f2228a8f" containerName="prom-label-proxy"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992920    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-thanos"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992930    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="prometheus"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992935    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="config-reloader"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.992940    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" containerName="kube-rbac-proxy-web"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.993141    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9"} err="failed to get container status \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": rpc error: code = NotFound desc = could not find container \"dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9\": container with ID starting with dceadbdfd9cc6ffa8e65bd16d870800b6878967c110f7716f6008552af403ca9 not found: ID does not exist"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.993173    2552 scope.go:117] "RemoveContainer" containerID="7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.993408    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327"} err="failed to get container status \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": rpc error: code = NotFound desc = could not find container \"7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327\": container with ID starting with 7a87b0304b29f75d4307de3eb190db6c1b6399f9d683caf5c19a9a3192de2327 not found: ID does not exist"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.993421    2552 scope.go:117] "RemoveContainer" containerID="af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.993647    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209"} err="failed to get container status \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": rpc error: code = NotFound desc = could not find container \"af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209\": container with ID starting with af26c593520f51bd7e6807e27365ebd5de9e1e5076fd583c1e7f516035304209 not found: ID does not exist"
Sep 10 12:12:25.993726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.993662    2552 scope.go:117] "RemoveContainer" containerID="8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"
Sep 10 12:12:25.994243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.993891    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a"} err="failed to get container status \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": rpc error: code = NotFound desc = could not find container \"8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a\": container with ID starting with 8befd90c640f5b8c1fae19f6436ce7a784f6d4d696d32f3f5373b488dc48574a not found: ID does not exist"
Sep 10 12:12:25.997931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:25.997915    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.000310 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.000293    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 10 12:12:26.000440 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.000409    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 10 12:12:26.000510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.000496    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 10 12:12:26.000706 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.000691    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 10 12:12:26.000775 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.000759    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 10 12:12:26.000901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.000887    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 10 12:12:26.001040 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001023    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-lm2ss\""
Sep 10 12:12:26.001290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001277    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 10 12:12:26.001391 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001374    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 10 12:12:26.001499 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001484    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 10 12:12:26.001555 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001530    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 10 12:12:26.001638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001622    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 10 12:12:26.001702 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001684    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 10 12:12:26.001770 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.001710    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-2754t0b8lut0t\""
Sep 10 12:12:26.002175 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.002156    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 10 12:12:26.004334 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.004319    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 10 12:12:26.006920 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.006901    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 10 12:12:26.015047 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.015030    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:12:26.063090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063061    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063090    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063108    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063133    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063213    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987b7840-2c80-4683-a8d4-fff29d1c838d-config-out\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063251    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063291    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063322    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063345    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063407    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063447    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063495    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063524    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063548    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063571    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063598    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-web-config\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063624    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-config\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063646    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987b7840-2c80-4683-a8d4-fff29d1c838d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063668    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.063908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.063703    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ffgf4\" (UniqueName: \"kubernetes.io/projected/987b7840-2c80-4683-a8d4-fff29d1c838d-kube-api-access-ffgf4\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164779 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164742    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164779 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164779    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164810    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164834    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164865    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164891    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164917    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164937    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164956    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-web-config\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.164991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.164981    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-config\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165002    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987b7840-2c80-4683-a8d4-fff29d1c838d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165026    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165064    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ffgf4\" (UniqueName: \"kubernetes.io/projected/987b7840-2c80-4683-a8d4-fff29d1c838d-kube-api-access-ffgf4\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165096    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165121    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165146    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165174    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165215    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987b7840-2c80-4683-a8d4-fff29d1c838d-config-out\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165324    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165490    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165606    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165675    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.165850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.165792    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.166635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.166605    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.166792 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.166766    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.167163 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.167136    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.168461 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.168384    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.168461 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.168404    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/987b7840-2c80-4683-a8d4-fff29d1c838d-config-out\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.168898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.168871    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.169031 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.169010    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.169309 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.169238    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.169873 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.169848    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.170386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.170357    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/987b7840-2c80-4683-a8d4-fff29d1c838d-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.170582 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.170562    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/987b7840-2c80-4683-a8d4-fff29d1c838d-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.170690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.170661    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.170728 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.170708    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.171017 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.171002    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-web-config\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.171176 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.171160    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/987b7840-2c80-4683-a8d4-fff29d1c838d-config\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.173703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.173687    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ffgf4\" (UniqueName: \"kubernetes.io/projected/987b7840-2c80-4683-a8d4-fff29d1c838d-kube-api-access-ffgf4\") pod \"prometheus-k8s-0\" (UID: \"987b7840-2c80-4683-a8d4-fff29d1c838d\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.308374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.308344    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:12:26.594406 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.594383    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 10 12:12:26.596018 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:12:26.595992    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod987b7840_2c80_4683_a8d4_fff29d1c838d.slice/crio-f6d8081f08ebe66e4ac5fd3036d463d9ddd7e4afa75759c7890cba19367fbef3 WatchSource:0}: Error finding container f6d8081f08ebe66e4ac5fd3036d463d9ddd7e4afa75759c7890cba19367fbef3: Status 404 returned error can't find the container with id f6d8081f08ebe66e4ac5fd3036d463d9ddd7e4afa75759c7890cba19367fbef3
Sep 10 12:12:26.941606 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.941569    2552 generic.go:356] "Generic (PLEG): container finished" podID="987b7840-2c80-4683-a8d4-fff29d1c838d" containerID="f52e9e03b147cc08eaa6f5008600bed9b86a44ab2c5e887409b74e36d3d6d8e9" exitCode=0
Sep 10 12:12:26.941763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.941660    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerDied","Data":"f52e9e03b147cc08eaa6f5008600bed9b86a44ab2c5e887409b74e36d3d6d8e9"}
Sep 10 12:12:26.941763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:26.941742    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerStarted","Data":"f6d8081f08ebe66e4ac5fd3036d463d9ddd7e4afa75759c7890cba19367fbef3"}
Sep 10 12:12:27.245185 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.245142    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3beb8dc2-7eb0-4f09-98fe-410ebd0097fb" path="/var/lib/kubelet/pods/3beb8dc2-7eb0-4f09-98fe-410ebd0097fb/volumes"
Sep 10 12:12:27.949964 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.949928    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerStarted","Data":"bf772dd6f281f8a5ff1b8f716e7a32a3f36c351d3870db039f0f993b25a922a4"}
Sep 10 12:12:27.950302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.949994    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerStarted","Data":"9d631102f61869d355a8f86faf6feeeb87225f73600deaa21cc58ea69b926e11"}
Sep 10 12:12:27.950302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.950021    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerStarted","Data":"ba958781aca517ea8bf5bca1bdaaf1204a0156c9db25f72cc38099c4b0776dcb"}
Sep 10 12:12:27.950302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.950044    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerStarted","Data":"9ac88b8c4cfd1a617b15e428a054badde42dbfeac246dd1fa9fb05af24013036"}
Sep 10 12:12:27.950302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.950066    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerStarted","Data":"225f0556f025d8f5ed915e6a8bfee74feb9575f8f7485499a561efe2e272a25c"}
Sep 10 12:12:27.950302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.950087    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"987b7840-2c80-4683-a8d4-fff29d1c838d","Type":"ContainerStarted","Data":"589bae517fd211d0ecbe81895dad66eb58c53fd3cba632ca50e9bb245bd56f1c"}
Sep 10 12:12:27.980648 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:27.980588    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=2.958149303 podStartE2EDuration="2.980567937s" totalImagesPullingTime="0s" totalInitContainerRuntime="22.418634ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:12:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:12:27.978937883 +0000 UTC m=+159.175453583" watchObservedRunningTime="2026-09-10 12:12:27.980567937 +0000 UTC m=+159.177083638"
Sep 10 12:12:31.309104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:12:31.309063    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:13:26.309202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:13:26.309170    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:13:26.324926 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:13:26.324900    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:13:27.173332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:13:27.173306    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 10 12:14:20.081826 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.081751    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5z67b"]
Sep 10 12:14:20.084809 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.084793    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.089065 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.089045    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 10 12:14:20.089175 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.089045    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-ctj44\""
Sep 10 12:14:20.089175 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.089045    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 10 12:14:20.096122 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.096103    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5z67b"]
Sep 10 12:14:20.204648 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.204612    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wg7m6\" (UniqueName: \"kubernetes.io/projected/8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8-kube-api-access-wg7m6\") pod \"cert-manager-55b4d954d5-5z67b\" (UID: \"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8\") " pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.204816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.204690    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8-bound-sa-token\") pod \"cert-manager-55b4d954d5-5z67b\" (UID: \"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8\") " pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.305229 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.305195    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8-bound-sa-token\") pod \"cert-manager-55b4d954d5-5z67b\" (UID: \"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8\") " pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.305229 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.305245    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wg7m6\" (UniqueName: \"kubernetes.io/projected/8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8-kube-api-access-wg7m6\") pod \"cert-manager-55b4d954d5-5z67b\" (UID: \"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8\") " pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.314114 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.314092    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wg7m6\" (UniqueName: \"kubernetes.io/projected/8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8-kube-api-access-wg7m6\") pod \"cert-manager-55b4d954d5-5z67b\" (UID: \"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8\") " pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.314227 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.314127    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8-bound-sa-token\") pod \"cert-manager-55b4d954d5-5z67b\" (UID: \"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8\") " pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.401131 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.401101    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-5z67b"
Sep 10 12:14:20.672876 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:20.672804    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-5z67b"]
Sep 10 12:14:20.675737 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:14:20.675712    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8daa2f69_04f6_4bc0_a87f_91dc91d0c5a8.slice/crio-61d5c8f01e8c9838b26d46fc1703dd9e411e896e22f8c35efa9b92421af08af6 WatchSource:0}: Error finding container 61d5c8f01e8c9838b26d46fc1703dd9e411e896e22f8c35efa9b92421af08af6: Status 404 returned error can't find the container with id 61d5c8f01e8c9838b26d46fc1703dd9e411e896e22f8c35efa9b92421af08af6
Sep 10 12:14:21.340200 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:21.340158    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5z67b" event={"ID":"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8","Type":"ContainerStarted","Data":"61d5c8f01e8c9838b26d46fc1703dd9e411e896e22f8c35efa9b92421af08af6"}
Sep 10 12:14:24.352755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:24.352722    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-5z67b" event={"ID":"8daa2f69-04f6-4bc0-a87f-91dc91d0c5a8","Type":"ContainerStarted","Data":"d4af6c84e87b746b51935a14dc0a926851134f6c8ed28622b7d4f771e907710c"}
Sep 10 12:14:24.399981 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:24.399931    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-5z67b" podStartSLOduration=1.649917301 podStartE2EDuration="4.399913675s" totalImagesPullingTime="2.749996374s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:14:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:14:24.398615861 +0000 UTC m=+275.595131550" watchObservedRunningTime="2026-09-10 12:14:24.399913675 +0000 UTC m=+275.596429375"
Sep 10 12:14:42.528959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.528923    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"]
Sep 10 12:14:42.532161 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.532145    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.542970 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.542945    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 12:14:42.543091 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.542953    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 12:14:42.543091 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.543035    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 12:14:42.545518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.545498    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 12:14:42.545644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.545522    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-wqs45\""
Sep 10 12:14:42.555337 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.555313    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hx78b\" (UniqueName: \"kubernetes.io/projected/6a637db7-ef43-4e36-92b1-5b66f960785c-kube-api-access-hx78b\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.555398 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.555365    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a637db7-ef43-4e36-92b1-5b66f960785c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.555398 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.555384    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a637db7-ef43-4e36-92b1-5b66f960785c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.555490 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.555441    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a637db7-ef43-4e36-92b1-5b66f960785c-tmp\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.555490 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.555461    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a637db7-ef43-4e36-92b1-5b66f960785c-manifests\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.562197 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.562168    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"]
Sep 10 12:14:42.655843 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.655800    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hx78b\" (UniqueName: \"kubernetes.io/projected/6a637db7-ef43-4e36-92b1-5b66f960785c-kube-api-access-hx78b\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.656057 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.655876    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a637db7-ef43-4e36-92b1-5b66f960785c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.656057 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.655901    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a637db7-ef43-4e36-92b1-5b66f960785c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.656057 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.655927    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a637db7-ef43-4e36-92b1-5b66f960785c-tmp\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.656057 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.655952    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a637db7-ef43-4e36-92b1-5b66f960785c-manifests\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.656392 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.656369    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/6a637db7-ef43-4e36-92b1-5b66f960785c-manifests\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.658313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.658284    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6a637db7-ef43-4e36-92b1-5b66f960785c-tmp\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.658415 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.658403    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/6a637db7-ef43-4e36-92b1-5b66f960785c-webhook-cert\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.658653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.658635    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/6a637db7-ef43-4e36-92b1-5b66f960785c-apiservice-cert\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.675106 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.675084    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hx78b\" (UniqueName: \"kubernetes.io/projected/6a637db7-ef43-4e36-92b1-5b66f960785c-kube-api-access-hx78b\") pod \"opendatahub-operator-controller-manager-5f75798bc7-rqm5n\" (UID: \"6a637db7-ef43-4e36-92b1-5b66f960785c\") " pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:42.841680 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:42.841592    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:43.117588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:43.117561    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"]
Sep 10 12:14:43.120551 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:14:43.120527    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6a637db7_ef43_4e36_92b1_5b66f960785c.slice/crio-26166f4cc85a455361725e12df0c4290bc524a12e36edb15f3897c266f7fabf8 WatchSource:0}: Error finding container 26166f4cc85a455361725e12df0c4290bc524a12e36edb15f3897c266f7fabf8: Status 404 returned error can't find the container with id 26166f4cc85a455361725e12df0c4290bc524a12e36edb15f3897c266f7fabf8
Sep 10 12:14:43.419856 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:43.419778    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n" event={"ID":"6a637db7-ef43-4e36-92b1-5b66f960785c","Type":"ContainerStarted","Data":"26166f4cc85a455361725e12df0c4290bc524a12e36edb15f3897c266f7fabf8"}
Sep 10 12:14:46.438541 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:46.438468    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n" event={"ID":"6a637db7-ef43-4e36-92b1-5b66f960785c","Type":"ContainerStarted","Data":"8434cfe6125256e30e9b1fc6b30e395c598921d0fb6a50ae84d21e1aedb67198"}
Sep 10 12:14:47.446728 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:47.446693    2552 generic.go:356] "Generic (PLEG): container finished" podID="6a637db7-ef43-4e36-92b1-5b66f960785c" containerID="8434cfe6125256e30e9b1fc6b30e395c598921d0fb6a50ae84d21e1aedb67198" exitCode=0
Sep 10 12:14:47.447062 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:47.446739    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n" event={"ID":"6a637db7-ef43-4e36-92b1-5b66f960785c","Type":"ContainerDied","Data":"8434cfe6125256e30e9b1fc6b30e395c598921d0fb6a50ae84d21e1aedb67198"}
Sep 10 12:14:48.451912 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:48.451883    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n" event={"ID":"6a637db7-ef43-4e36-92b1-5b66f960785c","Type":"ContainerStarted","Data":"96f86040404de066bdb710bfe051c7d725dccdbdf89b804dd032ef311dc581f5"}
Sep 10 12:14:48.452257 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:48.452095    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:48.482386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:48.482347    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n" podStartSLOduration=2.059371634 podStartE2EDuration="6.482318319s" totalImagesPullingTime="3.447889734s" totalInitContainerRuntime="975.056951ms" isStatefulPod=false podCreationTimestamp="2026-09-10 12:14:42 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:14:48.481216598 +0000 UTC m=+299.677732297" watchObservedRunningTime="2026-09-10 12:14:48.482318319 +0000 UTC m=+299.678834019"
Sep 10 12:14:49.162209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:49.162186    2552 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 12:14:59.458032 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.458006    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-5f75798bc7-rqm5n"
Sep 10 12:14:59.777598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.777525    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"]
Sep 10 12:14:59.779594 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.779578    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.782764 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.782737    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 12:14:59.783262 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.783236    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 12:14:59.783372 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.783271    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 12:14:59.783372 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.783331    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 12:14:59.783372 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.783336    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-jmg5p\""
Sep 10 12:14:59.783564 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.783336    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 12:14:59.795309 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.795292    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"]
Sep 10 12:14:59.883105 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.883072    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841d4a2-2f4c-4cb6-b420-6715cc0170be-metrics-cert\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.883105 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.883111    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6841d4a2-2f4c-4cb6-b420-6715cc0170be-manager-config\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.883250 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.883136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6841d4a2-2f4c-4cb6-b420-6715cc0170be-cert\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.883250 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.883178    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpw26\" (UniqueName: \"kubernetes.io/projected/6841d4a2-2f4c-4cb6-b420-6715cc0170be-kube-api-access-gpw26\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.984235 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.984205    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpw26\" (UniqueName: \"kubernetes.io/projected/6841d4a2-2f4c-4cb6-b420-6715cc0170be-kube-api-access-gpw26\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.984323 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.984263    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841d4a2-2f4c-4cb6-b420-6715cc0170be-metrics-cert\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.984323 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.984285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6841d4a2-2f4c-4cb6-b420-6715cc0170be-manager-config\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.984323 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.984310    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6841d4a2-2f4c-4cb6-b420-6715cc0170be-cert\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.985065 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.984992    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6841d4a2-2f4c-4cb6-b420-6715cc0170be-manager-config\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.986922 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.986902    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6841d4a2-2f4c-4cb6-b420-6715cc0170be-cert\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.986995 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.986922    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/6841d4a2-2f4c-4cb6-b420-6715cc0170be-metrics-cert\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:14:59.994298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:14:59.994277    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpw26\" (UniqueName: \"kubernetes.io/projected/6841d4a2-2f4c-4cb6-b420-6715cc0170be-kube-api-access-gpw26\") pod \"lws-controller-manager-74996bc6f6-rs762\" (UID: \"6841d4a2-2f4c-4cb6-b420-6715cc0170be\") " pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:15:00.088980 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:00.088923    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:15:00.363908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:00.363837    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"]
Sep 10 12:15:00.367751 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:15:00.367723    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6841d4a2_2f4c_4cb6_b420_6715cc0170be.slice/crio-2d3327dbce2cc4f65cb5286873bd5927607c2f206fae6f937dca0974962433fd WatchSource:0}: Error finding container 2d3327dbce2cc4f65cb5286873bd5927607c2f206fae6f937dca0974962433fd: Status 404 returned error can't find the container with id 2d3327dbce2cc4f65cb5286873bd5927607c2f206fae6f937dca0974962433fd
Sep 10 12:15:00.369799 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:00.369783    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:15:00.495016 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:00.494988    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762" event={"ID":"6841d4a2-2f4c-4cb6-b420-6715cc0170be","Type":"ContainerStarted","Data":"2d3327dbce2cc4f65cb5286873bd5927607c2f206fae6f937dca0974962433fd"}
Sep 10 12:15:01.867362 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.867332    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"]
Sep 10 12:15:01.876882 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.876849    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:01.879938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.879913    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nqqnz\""
Sep 10 12:15:01.880089 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.879913    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 12:15:01.880962 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.880940    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 12:15:01.880962 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.880941    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 12:15:01.881143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.880945    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 12:15:01.887308 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.887284    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"]
Sep 10 12:15:01.899164 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.899139    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kp8gv\" (UniqueName: \"kubernetes.io/projected/738e179c-8351-43a8-9f34-d131f8bed321-kube-api-access-kp8gv\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:01.899278 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.899211    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/738e179c-8351-43a8-9f34-d131f8bed321-tmp\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:01.899339 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:01.899295    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/738e179c-8351-43a8-9f34-d131f8bed321-tls-certs\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.000776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.000738    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/738e179c-8351-43a8-9f34-d131f8bed321-tmp\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.000959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.000800    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/738e179c-8351-43a8-9f34-d131f8bed321-tls-certs\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.000959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.000856    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kp8gv\" (UniqueName: \"kubernetes.io/projected/738e179c-8351-43a8-9f34-d131f8bed321-kube-api-access-kp8gv\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.001086 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:15:02.000984    2552 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 10 12:15:02.001086 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:15:02.001059    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/738e179c-8351-43a8-9f34-d131f8bed321-tls-certs podName:738e179c-8351-43a8-9f34-d131f8bed321 nodeName:}" failed. No retries permitted until 2026-09-10 12:15:02.501035618 +0000 UTC m=+313.697551332 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/738e179c-8351-43a8-9f34-d131f8bed321-tls-certs") pod "kube-auth-proxy-688cb69f77-h56qn" (UID: "738e179c-8351-43a8-9f34-d131f8bed321") : secret "kube-auth-proxy-tls" not found
Sep 10 12:15:02.003391 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.003362    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/738e179c-8351-43a8-9f34-d131f8bed321-tmp\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.012829 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.012800    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kp8gv\" (UniqueName: \"kubernetes.io/projected/738e179c-8351-43a8-9f34-d131f8bed321-kube-api-access-kp8gv\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.505501 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.505417    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/738e179c-8351-43a8-9f34-d131f8bed321-tls-certs\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.508040 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.508010    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/738e179c-8351-43a8-9f34-d131f8bed321-tls-certs\") pod \"kube-auth-proxy-688cb69f77-h56qn\" (UID: \"738e179c-8351-43a8-9f34-d131f8bed321\") " pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:02.789508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:02.789378    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"
Sep 10 12:15:03.059400 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:03.059331    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-688cb69f77-h56qn"]
Sep 10 12:15:03.062359 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:15:03.062325    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod738e179c_8351_43a8_9f34_d131f8bed321.slice/crio-9c8c3ab60bea1a360b8448c6d841f340c234afa72293c5d8367fa7e9cbe18b30 WatchSource:0}: Error finding container 9c8c3ab60bea1a360b8448c6d841f340c234afa72293c5d8367fa7e9cbe18b30: Status 404 returned error can't find the container with id 9c8c3ab60bea1a360b8448c6d841f340c234afa72293c5d8367fa7e9cbe18b30
Sep 10 12:15:03.512936 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:03.512898    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn" event={"ID":"738e179c-8351-43a8-9f34-d131f8bed321","Type":"ContainerStarted","Data":"9c8c3ab60bea1a360b8448c6d841f340c234afa72293c5d8367fa7e9cbe18b30"}
Sep 10 12:15:03.514403 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:03.514381    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762" event={"ID":"6841d4a2-2f4c-4cb6-b420-6715cc0170be","Type":"ContainerStarted","Data":"54bde84944281a8804a5851b49d32ae61f3bab091c776ffb0e456e6755e66920"}
Sep 10 12:15:03.534081 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:03.534032    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762" podStartSLOduration=2.429933961 podStartE2EDuration="4.534018715s" totalImagesPullingTime="2.104084754s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:14:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:03.533065214 +0000 UTC m=+314.729580915" watchObservedRunningTime="2026-09-10 12:15:03.534018715 +0000 UTC m=+314.730534411"
Sep 10 12:15:04.518394 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:04.518363    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:15:06.528242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:06.528162    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn" event={"ID":"738e179c-8351-43a8-9f34-d131f8bed321","Type":"ContainerStarted","Data":"b63e28d0562111b45735439d0f8501e3d85c9771c5ced9a8f2fcddc3968bc78b"}
Sep 10 12:15:06.548846 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:06.548795    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-688cb69f77-h56qn" podStartSLOduration=2.488529751 podStartE2EDuration="5.548776654s" totalImagesPullingTime="3.060246903s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:06.547901736 +0000 UTC m=+317.744417437" watchObservedRunningTime="2026-09-10 12:15:06.548776654 +0000 UTC m=+317.745292353"
Sep 10 12:15:15.525244 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:15.525209    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-74996bc6f6-rs762"
Sep 10 12:15:24.691534 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.691497    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"]
Sep 10 12:15:24.697615 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.697599    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:24.704585 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.704565    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 10 12:15:24.704772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.704755    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-nj9l7\""
Sep 10 12:15:24.792275 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.792237    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f55n\" (UniqueName: \"kubernetes.io/projected/6b3b51e6-635c-4332-aff7-76fdd9b8e822-kube-api-access-5f55n\") pod \"llmisvc-controller-manager-777c6bd7d5-bl6x9\" (UID: \"6b3b51e6-635c-4332-aff7-76fdd9b8e822\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:24.792538 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.792341    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b3b51e6-635c-4332-aff7-76fdd9b8e822-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bl6x9\" (UID: \"6b3b51e6-635c-4332-aff7-76fdd9b8e822\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:24.794413 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.794379    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"]
Sep 10 12:15:24.893444 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.893387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b3b51e6-635c-4332-aff7-76fdd9b8e822-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bl6x9\" (UID: \"6b3b51e6-635c-4332-aff7-76fdd9b8e822\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:24.893604 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.893470    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f55n\" (UniqueName: \"kubernetes.io/projected/6b3b51e6-635c-4332-aff7-76fdd9b8e822-kube-api-access-5f55n\") pod \"llmisvc-controller-manager-777c6bd7d5-bl6x9\" (UID: \"6b3b51e6-635c-4332-aff7-76fdd9b8e822\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:24.895989 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.895969    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6b3b51e6-635c-4332-aff7-76fdd9b8e822-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-bl6x9\" (UID: \"6b3b51e6-635c-4332-aff7-76fdd9b8e822\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:24.910060 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:24.910027    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f55n\" (UniqueName: \"kubernetes.io/projected/6b3b51e6-635c-4332-aff7-76fdd9b8e822-kube-api-access-5f55n\") pod \"llmisvc-controller-manager-777c6bd7d5-bl6x9\" (UID: \"6b3b51e6-635c-4332-aff7-76fdd9b8e822\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:25.007336 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:25.007263    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:25.289154 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:25.289069    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"]
Sep 10 12:15:25.293579 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:15:25.293551    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod6b3b51e6_635c_4332_aff7_76fdd9b8e822.slice/crio-352f232265833208b3b7dc19df7562916794f932a111f395f8237e7e094ef0db WatchSource:0}: Error finding container 352f232265833208b3b7dc19df7562916794f932a111f395f8237e7e094ef0db: Status 404 returned error can't find the container with id 352f232265833208b3b7dc19df7562916794f932a111f395f8237e7e094ef0db
Sep 10 12:15:25.596743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:25.596661    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9" event={"ID":"6b3b51e6-635c-4332-aff7-76fdd9b8e822","Type":"ContainerStarted","Data":"352f232265833208b3b7dc19df7562916794f932a111f395f8237e7e094ef0db"}
Sep 10 12:15:28.611085 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:28.611047    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9" event={"ID":"6b3b51e6-635c-4332-aff7-76fdd9b8e822","Type":"ContainerStarted","Data":"779268c21b648a61f84982e23b980bfda6c5f334e937ad2d8e3023ac9721b1d8"}
Sep 10 12:15:28.611510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:28.611162    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:15:28.637928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:28.637885    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9" podStartSLOduration=2.166350653 podStartE2EDuration="4.637871931s" totalImagesPullingTime="2.471521278s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:15:24 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:15:28.63753372 +0000 UTC m=+339.834049421" watchObservedRunningTime="2026-09-10 12:15:28.637871931 +0000 UTC m=+339.834387630"
Sep 10 12:15:59.619860 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:15:59.619829    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-bl6x9"
Sep 10 12:17:07.952892 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.952855    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"]
Sep 10 12:17:07.956067 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.956051    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:07.959146 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.959120    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 12:17:07.959286 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.959236    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 12:17:07.960220 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.960202    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 12:17:07.960275 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.960228    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 12:17:07.960275 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.960231    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-pz4lb\""
Sep 10 12:17:07.965331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:07.965308    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"]
Sep 10 12:17:08.094743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.094705    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj594\" (UniqueName: \"kubernetes.io/projected/4e39e5fa-2a86-4f4e-86c0-059717f4187e-kube-api-access-dj594\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.094911 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.094764    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e39e5fa-2a86-4f4e-86c0-059717f4187e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.094911 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.094863    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e39e5fa-2a86-4f4e-86c0-059717f4187e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.196204 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.196164    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e39e5fa-2a86-4f4e-86c0-059717f4187e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.196341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.196218    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj594\" (UniqueName: \"kubernetes.io/projected/4e39e5fa-2a86-4f4e-86c0-059717f4187e-kube-api-access-dj594\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.196341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.196256    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e39e5fa-2a86-4f4e-86c0-059717f4187e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.196341 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:17:08.196319    2552 secret.go:189] Couldn't get secret kuadrant-system/plugin-serving-cert: secret "plugin-serving-cert" not found
Sep 10 12:17:08.196463 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:17:08.196389    2552 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4e39e5fa-2a86-4f4e-86c0-059717f4187e-plugin-serving-cert podName:4e39e5fa-2a86-4f4e-86c0-059717f4187e nodeName:}" failed. No retries permitted until 2026-09-10 12:17:08.696369879 +0000 UTC m=+439.892885557 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugin-serving-cert" (UniqueName: "kubernetes.io/secret/4e39e5fa-2a86-4f4e-86c0-059717f4187e-plugin-serving-cert") pod "kuadrant-console-plugin-fbdd4f445-xtmsn" (UID: "4e39e5fa-2a86-4f4e-86c0-059717f4187e") : secret "plugin-serving-cert" not found
Sep 10 12:17:08.196913 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.196896    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/4e39e5fa-2a86-4f4e-86c0-059717f4187e-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.206020 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.205964    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj594\" (UniqueName: \"kubernetes.io/projected/4e39e5fa-2a86-4f4e-86c0-059717f4187e-kube-api-access-dj594\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.699267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.699225    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e39e5fa-2a86-4f4e-86c0-059717f4187e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.701830 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.701807    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/4e39e5fa-2a86-4f4e-86c0-059717f4187e-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-xtmsn\" (UID: \"4e39e5fa-2a86-4f4e-86c0-059717f4187e\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:08.865017 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:08.864983    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"
Sep 10 12:17:09.137558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:09.137534    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn"]
Sep 10 12:17:09.139516 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:17:09.139492    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod4e39e5fa_2a86_4f4e_86c0_059717f4187e.slice/crio-9aaea5bbdf30e04b8ed896c8177ea80261ddc549cf9253f510fcd97098cb639a WatchSource:0}: Error finding container 9aaea5bbdf30e04b8ed896c8177ea80261ddc549cf9253f510fcd97098cb639a: Status 404 returned error can't find the container with id 9aaea5bbdf30e04b8ed896c8177ea80261ddc549cf9253f510fcd97098cb639a
Sep 10 12:17:09.990539 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:09.990508    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn" event={"ID":"4e39e5fa-2a86-4f4e-86c0-059717f4187e","Type":"ContainerStarted","Data":"9aaea5bbdf30e04b8ed896c8177ea80261ddc549cf9253f510fcd97098cb639a"}
Sep 10 12:17:16.021752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:16.021713    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn" event={"ID":"4e39e5fa-2a86-4f4e-86c0-059717f4187e","Type":"ContainerStarted","Data":"920c9d5ca39e9dd2517b5e8239a0e3efeb19c97c8e3a29c7280135564f07f6f5"}
Sep 10 12:17:51.946914 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:51.946860    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-xtmsn" podStartSLOduration=38.642170899 podStartE2EDuration="44.946841609s" totalImagesPullingTime="6.30467071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:17:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:17:16.066456593 +0000 UTC m=+447.262972283" watchObservedRunningTime="2026-09-10 12:17:51.946841609 +0000 UTC m=+483.143357309"
Sep 10 12:17:51.947739 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:51.947722    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:17:51.950983 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:51.950955    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:51.954100 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:51.954075    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 12:17:51.970298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:51.970277    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:17:52.032045 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.032020    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:17:52.040883 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.040859    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfab91b6-0ccc-4e12-bf0b-cc6b9265b791-config-file\") pod \"limitador-limitador-84994f7ffc-vf92m\" (UID: \"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:52.040971 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.040957    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/cfab91b6-0ccc-4e12-bf0b-cc6b9265b791-kube-api-access-zgbh2\") pod \"limitador-limitador-84994f7ffc-vf92m\" (UID: \"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:52.141498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.141474    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/cfab91b6-0ccc-4e12-bf0b-cc6b9265b791-kube-api-access-zgbh2\") pod \"limitador-limitador-84994f7ffc-vf92m\" (UID: \"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:52.141576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.141504    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfab91b6-0ccc-4e12-bf0b-cc6b9265b791-config-file\") pod \"limitador-limitador-84994f7ffc-vf92m\" (UID: \"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:52.142029 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.142014    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cfab91b6-0ccc-4e12-bf0b-cc6b9265b791-config-file\") pod \"limitador-limitador-84994f7ffc-vf92m\" (UID: \"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:52.150292 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.150264    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zgbh2\" (UniqueName: \"kubernetes.io/projected/cfab91b6-0ccc-4e12-bf0b-cc6b9265b791-kube-api-access-zgbh2\") pod \"limitador-limitador-84994f7ffc-vf92m\" (UID: \"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:52.260727 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.260670    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:52.533940 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.533919    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:17:52.535939 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:17:52.535913    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcfab91b6_0ccc_4e12_bf0b_cc6b9265b791.slice/crio-887506a43d877af9706a94deab6fbaebe648b7ce541e499fdefb92e52e0987a4 WatchSource:0}: Error finding container 887506a43d877af9706a94deab6fbaebe648b7ce541e499fdefb92e52e0987a4: Status 404 returned error can't find the container with id 887506a43d877af9706a94deab6fbaebe648b7ce541e499fdefb92e52e0987a4
Sep 10 12:17:52.779119 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.779094    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-59c974bd48-ztqbs"]
Sep 10 12:17:52.783538 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.783495    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ztqbs"
Sep 10 12:17:52.786531 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.786513    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-b9vcl\""
Sep 10 12:17:52.787121 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.787102    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ztqbs"]
Sep 10 12:17:52.846794 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.846767    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxdp7\" (UniqueName: \"kubernetes.io/projected/7cf336ee-5bc1-4a84-a5eb-fc451382daaf-kube-api-access-vxdp7\") pod \"authorino-59c974bd48-ztqbs\" (UID: \"7cf336ee-5bc1-4a84-a5eb-fc451382daaf\") " pod="kuadrant-system/authorino-59c974bd48-ztqbs"
Sep 10 12:17:52.947508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.947481    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxdp7\" (UniqueName: \"kubernetes.io/projected/7cf336ee-5bc1-4a84-a5eb-fc451382daaf-kube-api-access-vxdp7\") pod \"authorino-59c974bd48-ztqbs\" (UID: \"7cf336ee-5bc1-4a84-a5eb-fc451382daaf\") " pod="kuadrant-system/authorino-59c974bd48-ztqbs"
Sep 10 12:17:52.955906 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:52.955883    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxdp7\" (UniqueName: \"kubernetes.io/projected/7cf336ee-5bc1-4a84-a5eb-fc451382daaf-kube-api-access-vxdp7\") pod \"authorino-59c974bd48-ztqbs\" (UID: \"7cf336ee-5bc1-4a84-a5eb-fc451382daaf\") " pod="kuadrant-system/authorino-59c974bd48-ztqbs"
Sep 10 12:17:53.093234 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:53.093010    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ztqbs"
Sep 10 12:17:53.165638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:53.165604    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m" event={"ID":"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791","Type":"ContainerStarted","Data":"887506a43d877af9706a94deab6fbaebe648b7ce541e499fdefb92e52e0987a4"}
Sep 10 12:17:53.374067 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:53.374038    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ztqbs"]
Sep 10 12:17:53.377219 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:17:53.377190    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7cf336ee_5bc1_4a84_a5eb_fc451382daaf.slice/crio-8c9bea19043135e69a46890a434217fe4bfa217140d68522ab353f68f9cb1e37 WatchSource:0}: Error finding container 8c9bea19043135e69a46890a434217fe4bfa217140d68522ab353f68f9cb1e37: Status 404 returned error can't find the container with id 8c9bea19043135e69a46890a434217fe4bfa217140d68522ab353f68f9cb1e37
Sep 10 12:17:54.171398 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:54.171359    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ztqbs" event={"ID":"7cf336ee-5bc1-4a84-a5eb-fc451382daaf","Type":"ContainerStarted","Data":"8c9bea19043135e69a46890a434217fe4bfa217140d68522ab353f68f9cb1e37"}
Sep 10 12:17:55.178348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:55.178308    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m" event={"ID":"cfab91b6-0ccc-4e12-bf0b-cc6b9265b791","Type":"ContainerStarted","Data":"06a6e72b7108d298098252d6b8ee633fc6a2094589842d2d4edbb22d2b0d1182"}
Sep 10 12:17:55.178810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:55.178390    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:17:55.197291 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:55.197232    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m" podStartSLOduration=2.181181259 podStartE2EDuration="4.197212411s" totalImagesPullingTime="2.016031152s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:17:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:17:55.1958213 +0000 UTC m=+486.392336999" watchObservedRunningTime="2026-09-10 12:17:55.197212411 +0000 UTC m=+486.393728112"
Sep 10 12:17:56.184515 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:56.184461    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ztqbs" event={"ID":"7cf336ee-5bc1-4a84-a5eb-fc451382daaf","Type":"ContainerStarted","Data":"07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771"}
Sep 10 12:17:56.203797 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:17:56.203746    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-59c974bd48-ztqbs" podStartSLOduration=2.077773154 podStartE2EDuration="4.203731246s" totalImagesPullingTime="2.125958092s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:17:52 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:17:56.202676612 +0000 UTC m=+487.399192312" watchObservedRunningTime="2026-09-10 12:17:56.203731246 +0000 UTC m=+487.400246945"
Sep 10 12:18:06.185494 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:06.185465    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-vf92m"
Sep 10 12:18:20.008319 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.008281    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-986d6d768-kx6zp"]
Sep 10 12:18:20.011829 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.011804    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.015109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.015086    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 12:18:20.022704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.022683    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-986d6d768-kx6zp"]
Sep 10 12:18:20.137350 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.137324    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tnt92\" (UniqueName: \"kubernetes.io/projected/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-kube-api-access-tnt92\") pod \"authorino-986d6d768-kx6zp\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") " pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.137474 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.137356    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-tls-cert\") pod \"authorino-986d6d768-kx6zp\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") " pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.238555 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.238532    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tnt92\" (UniqueName: \"kubernetes.io/projected/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-kube-api-access-tnt92\") pod \"authorino-986d6d768-kx6zp\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") " pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.238656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.238561    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-tls-cert\") pod \"authorino-986d6d768-kx6zp\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") " pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.241203 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.241177    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-tls-cert\") pod \"authorino-986d6d768-kx6zp\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") " pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.247322 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.247301    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tnt92\" (UniqueName: \"kubernetes.io/projected/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-kube-api-access-tnt92\") pod \"authorino-986d6d768-kx6zp\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") " pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.321042 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.320993    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:18:20.606206 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:20.606185    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-986d6d768-kx6zp"]
Sep 10 12:18:20.608460 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:18:20.608416    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda7fe4f96_cc2a_4f9b_a603_7a3f791bd0b9.slice/crio-457100f0a8d0b8c8884cfa2fe12de04c92d63c2e29132cfa01791a04928ad4c6 WatchSource:0}: Error finding container 457100f0a8d0b8c8884cfa2fe12de04c92d63c2e29132cfa01791a04928ad4c6: Status 404 returned error can't find the container with id 457100f0a8d0b8c8884cfa2fe12de04c92d63c2e29132cfa01791a04928ad4c6
Sep 10 12:18:21.287537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.287458    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-986d6d768-kx6zp" event={"ID":"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9","Type":"ContainerStarted","Data":"768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d"}
Sep 10 12:18:21.287537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.287502    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-986d6d768-kx6zp" event={"ID":"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9","Type":"ContainerStarted","Data":"457100f0a8d0b8c8884cfa2fe12de04c92d63c2e29132cfa01791a04928ad4c6"}
Sep 10 12:18:21.306187 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.306141    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-986d6d768-kx6zp" podStartSLOduration=1.8786083599999999 podStartE2EDuration="2.306127178s" totalImagesPullingTime="427.518818ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:18:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:18:21.305163595 +0000 UTC m=+512.501679295" watchObservedRunningTime="2026-09-10 12:18:21.306127178 +0000 UTC m=+512.502642878"
Sep 10 12:18:21.336014 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.335987    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ztqbs"]
Sep 10 12:18:21.336159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.336139    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-59c974bd48-ztqbs" podUID="7cf336ee-5bc1-4a84-a5eb-fc451382daaf" containerName="authorino" containerID="cri-o://07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771" gracePeriod=30
Sep 10 12:18:21.572559 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.572534    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ztqbs"
Sep 10 12:18:21.650055 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.650030    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vxdp7\" (UniqueName: \"kubernetes.io/projected/7cf336ee-5bc1-4a84-a5eb-fc451382daaf-kube-api-access-vxdp7\") pod \"7cf336ee-5bc1-4a84-a5eb-fc451382daaf\" (UID: \"7cf336ee-5bc1-4a84-a5eb-fc451382daaf\") "
Sep 10 12:18:21.652178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.652153    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7cf336ee-5bc1-4a84-a5eb-fc451382daaf-kube-api-access-vxdp7" (OuterVolumeSpecName: "kube-api-access-vxdp7") pod "7cf336ee-5bc1-4a84-a5eb-fc451382daaf" (UID: "7cf336ee-5bc1-4a84-a5eb-fc451382daaf"). InnerVolumeSpecName "kube-api-access-vxdp7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:18:21.750732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:21.750710    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxdp7\" (UniqueName: \"kubernetes.io/projected/7cf336ee-5bc1-4a84-a5eb-fc451382daaf-kube-api-access-vxdp7\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:18:22.293495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.293464    2552 generic.go:356] "Generic (PLEG): container finished" podID="7cf336ee-5bc1-4a84-a5eb-fc451382daaf" containerID="07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771" exitCode=0
Sep 10 12:18:22.293928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.293505    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ztqbs" event={"ID":"7cf336ee-5bc1-4a84-a5eb-fc451382daaf","Type":"ContainerDied","Data":"07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771"}
Sep 10 12:18:22.293928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.293548    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-59c974bd48-ztqbs" event={"ID":"7cf336ee-5bc1-4a84-a5eb-fc451382daaf","Type":"ContainerDied","Data":"8c9bea19043135e69a46890a434217fe4bfa217140d68522ab353f68f9cb1e37"}
Sep 10 12:18:22.293928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.293565    2552 scope.go:117] "RemoveContainer" containerID="07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771"
Sep 10 12:18:22.293928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.293517    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-59c974bd48-ztqbs"
Sep 10 12:18:22.301871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.301856    2552 scope.go:117] "RemoveContainer" containerID="07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771"
Sep 10 12:18:22.302187 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:18:22.302166    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771\": container with ID starting with 07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771 not found: ID does not exist" containerID="07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771"
Sep 10 12:18:22.302245 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.302195    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771"} err="failed to get container status \"07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771\": rpc error: code = NotFound desc = could not find container \"07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771\": container with ID starting with 07737a148fe9d2b1c4235a68ad818f004276162f79daab86e2573ffa198b8771 not found: ID does not exist"
Sep 10 12:18:22.314821 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.314798    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ztqbs"]
Sep 10 12:18:22.317100 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:22.317078    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-59c974bd48-ztqbs"]
Sep 10 12:18:23.241339 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:18:23.241308    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7cf336ee-5bc1-4a84-a5eb-fc451382daaf" path="/var/lib/kubelet/pods/7cf336ee-5bc1-4a84-a5eb-fc451382daaf/volumes"
Sep 10 12:19:01.602455 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:19:01.602397    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:19:23.863647 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:19:23.863613    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:19:33.143122 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:19:33.143086    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:19:41.477458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:19:41.477178    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:19:51.242486 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:19:51.242458    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:20:04.473143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.473104    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"]
Sep 10 12:20:04.475545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.473617    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7cf336ee-5bc1-4a84-a5eb-fc451382daaf" containerName="authorino"
Sep 10 12:20:04.475545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.473633    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7cf336ee-5bc1-4a84-a5eb-fc451382daaf" containerName="authorino"
Sep 10 12:20:04.475545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.473705    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7cf336ee-5bc1-4a84-a5eb-fc451382daaf" containerName="authorino"
Sep 10 12:20:04.476692 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.476677    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.482788 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.482765    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-x8ww2\""
Sep 10 12:20:04.482928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.482765    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 12:20:04.482928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.482770    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 12:20:04.482928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.482769    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 12:20:04.503760 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.503731    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"]
Sep 10 12:20:04.593383 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.593341    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.593383 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.593385    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bdf325-2124-490f-a1a5-4e3bab3ac86c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.593625 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.593413    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vsbl7\" (UniqueName: \"kubernetes.io/projected/03bdf325-2124-490f-a1a5-4e3bab3ac86c-kube-api-access-vsbl7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.593625 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.593502    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.593625 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.593593    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.593738 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.593626    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.694852 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.694814    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.694852 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.694855    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.695107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.694874    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.695107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.694901    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bdf325-2124-490f-a1a5-4e3bab3ac86c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.695107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.694927    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vsbl7\" (UniqueName: \"kubernetes.io/projected/03bdf325-2124-490f-a1a5-4e3bab3ac86c-kube-api-access-vsbl7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.695107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.694981    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.695311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.695279    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.695372 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.695339    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.695412 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.695366    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.697464 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.697421    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/03bdf325-2124-490f-a1a5-4e3bab3ac86c-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.697831 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.697806    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/03bdf325-2124-490f-a1a5-4e3bab3ac86c-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.703726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.703702    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vsbl7\" (UniqueName: \"kubernetes.io/projected/03bdf325-2124-490f-a1a5-4e3bab3ac86c-kube-api-access-vsbl7\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft\" (UID: \"03bdf325-2124-490f-a1a5-4e3bab3ac86c\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:04.786678 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:04.786574    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:05.062657 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:05.062632    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"]
Sep 10 12:20:05.064739 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:20:05.064713    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03bdf325_2124_490f_a1a5_4e3bab3ac86c.slice/crio-763084c47b434c5afb6c51cfd914cb947769f3ac08f4a2d1a86d9d3d70af58fc WatchSource:0}: Error finding container 763084c47b434c5afb6c51cfd914cb947769f3ac08f4a2d1a86d9d3d70af58fc: Status 404 returned error can't find the container with id 763084c47b434c5afb6c51cfd914cb947769f3ac08f4a2d1a86d9d3d70af58fc
Sep 10 12:20:05.065977 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:05.065958    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:20:05.724569 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:05.724534    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft" event={"ID":"03bdf325-2124-490f-a1a5-4e3bab3ac86c","Type":"ContainerStarted","Data":"763084c47b434c5afb6c51cfd914cb947769f3ac08f4a2d1a86d9d3d70af58fc"}
Sep 10 12:20:05.844267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:05.844224    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:20:06.731544 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:06.731504    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft" event={"ID":"03bdf325-2124-490f-a1a5-4e3bab3ac86c","Type":"ContainerStarted","Data":"60bf582a27f7038d14bc8fb710a2e1ceb261493fbdc825d7879c8e75bd2bd087"}
Sep 10 12:20:06.753123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:06.753058    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft" podStartSLOduration=1.630785254 podStartE2EDuration="2.753039343s" totalImagesPullingTime="1.122254089s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:20:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:20:06.751959777 +0000 UTC m=+617.948475491" watchObservedRunningTime="2026-09-10 12:20:06.753039343 +0000 UTC m=+617.949555043"
Sep 10 12:20:14.787159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:14.787073    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:14.787159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:14.787130    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:14.805147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:14.805112    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:15.780949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:15.780918    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft"
Sep 10 12:20:19.042950 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:19.042906    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:20:29.051626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.051593    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"]
Sep 10 12:20:29.057675 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.057655    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.060624 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.060601    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 12:20:29.068475 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.068447    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"]
Sep 10 12:20:29.117970 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.117928    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.117970 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.117970    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.118236 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.117996    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e689f2-45b4-4bb9-bc2e-66dd738acae0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.118236 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.118078    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.118236 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.118128    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.118236 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.118189    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mp4zl\" (UniqueName: \"kubernetes.io/projected/01e689f2-45b4-4bb9-bc2e-66dd738acae0-kube-api-access-mp4zl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219186 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219141    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219186 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219184    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219210    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e689f2-45b4-4bb9-bc2e-66dd738acae0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219329    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219372    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219452    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mp4zl\" (UniqueName: \"kubernetes.io/projected/01e689f2-45b4-4bb9-bc2e-66dd738acae0-kube-api-access-mp4zl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219627    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219682    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.219752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.219711    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.221719 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.221690    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/01e689f2-45b4-4bb9-bc2e-66dd738acae0-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.221977 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.221960    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/01e689f2-45b4-4bb9-bc2e-66dd738acae0-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.229621 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.229594    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mp4zl\" (UniqueName: \"kubernetes.io/projected/01e689f2-45b4-4bb9-bc2e-66dd738acae0-kube-api-access-mp4zl\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-m28qc\" (UID: \"01e689f2-45b4-4bb9-bc2e-66dd738acae0\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.366534 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.366505    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:29.644395 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.644370    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"]
Sep 10 12:20:29.646870 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:20:29.646835    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod01e689f2_45b4_4bb9_bc2e_66dd738acae0.slice/crio-828e385f5c63d8e44cd02c68ccf0a2cc086e411eb9137725cfb9199a7e9e1ce3 WatchSource:0}: Error finding container 828e385f5c63d8e44cd02c68ccf0a2cc086e411eb9137725cfb9199a7e9e1ce3: Status 404 returned error can't find the container with id 828e385f5c63d8e44cd02c68ccf0a2cc086e411eb9137725cfb9199a7e9e1ce3
Sep 10 12:20:29.827108 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:29.827071    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc" event={"ID":"01e689f2-45b4-4bb9-bc2e-66dd738acae0","Type":"ContainerStarted","Data":"828e385f5c63d8e44cd02c68ccf0a2cc086e411eb9137725cfb9199a7e9e1ce3"}
Sep 10 12:20:30.832712 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:30.832673    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc" event={"ID":"01e689f2-45b4-4bb9-bc2e-66dd738acae0","Type":"ContainerStarted","Data":"bf2cfc4734304a748957a651361466771e9dd7eaffe048ceece6c1d2b3c01f64"}
Sep 10 12:20:30.851139 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:30.851084    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc" podStartSLOduration=1.657927195 podStartE2EDuration="1.851064994s" totalImagesPullingTime="193.137799ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:20:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:20:30.849861784 +0000 UTC m=+642.046377484" watchObservedRunningTime="2026-09-10 12:20:30.851064994 +0000 UTC m=+642.047580693"
Sep 10 12:20:31.040846 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:31.040807    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:20:39.366639 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:39.366595    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:39.367213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:39.366655    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:39.379597 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:39.379568    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:20:39.882911 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:20:39.882875    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-m28qc"
Sep 10 12:21:55.312929 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.312846    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-986d6d768-kx6zp"]
Sep 10 12:21:55.313342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.313031    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-986d6d768-kx6zp" podUID="a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" containerName="authorino" containerID="cri-o://768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d" gracePeriod=30
Sep 10 12:21:55.557253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.557231    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:21:55.648597 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.648574    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tnt92\" (UniqueName: \"kubernetes.io/projected/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-kube-api-access-tnt92\") pod \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") "
Sep 10 12:21:55.648697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.648620    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-tls-cert\") pod \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\" (UID: \"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9\") "
Sep 10 12:21:55.650719 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.650696    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-kube-api-access-tnt92" (OuterVolumeSpecName: "kube-api-access-tnt92") pod "a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" (UID: "a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9"). InnerVolumeSpecName "kube-api-access-tnt92". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:21:55.657925 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.657898    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" (UID: "a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:21:55.749545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.749523    2552 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-tls-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:21:55.749545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:55.749542    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tnt92\" (UniqueName: \"kubernetes.io/projected/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9-kube-api-access-tnt92\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:21:56.191644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.191610    2552 generic.go:356] "Generic (PLEG): container finished" podID="a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" containerID="768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d" exitCode=0
Sep 10 12:21:56.191796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.191654    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-986d6d768-kx6zp"
Sep 10 12:21:56.191796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.191671    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-986d6d768-kx6zp" event={"ID":"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9","Type":"ContainerDied","Data":"768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d"}
Sep 10 12:21:56.191796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.191704    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-986d6d768-kx6zp" event={"ID":"a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9","Type":"ContainerDied","Data":"457100f0a8d0b8c8884cfa2fe12de04c92d63c2e29132cfa01791a04928ad4c6"}
Sep 10 12:21:56.191796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.191720    2552 scope.go:117] "RemoveContainer" containerID="768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d"
Sep 10 12:21:56.202354 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.202338    2552 scope.go:117] "RemoveContainer" containerID="768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d"
Sep 10 12:21:56.202697 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:21:56.202678    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d\": container with ID starting with 768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d not found: ID does not exist" containerID="768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d"
Sep 10 12:21:56.202772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.202705    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d"} err="failed to get container status \"768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d\": rpc error: code = NotFound desc = could not find container \"768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d\": container with ID starting with 768b84a945a3b1bc3ddd3792ee94b1f4afdf764f22d3c8dcf84061752da3539d not found: ID does not exist"
Sep 10 12:21:56.213890 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.213871    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-986d6d768-kx6zp"]
Sep 10 12:21:56.220210 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:56.220191    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-986d6d768-kx6zp"]
Sep 10 12:21:57.242644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:21:57.242591    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" path="/var/lib/kubelet/pods/a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9/volumes"
Sep 10 12:22:13.969394 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:13.969359    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-140247792414384"]
Sep 10 12:22:13.969900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:13.969673    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" containerName="authorino"
Sep 10 12:22:13.969900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:13.969685    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" containerName="authorino"
Sep 10 12:22:13.969900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:13.969745    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a7fe4f96-cc2a-4f9b-a603-7a3f791bd0b9" containerName="authorino"
Sep 10 12:22:13.972502 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:13.972487    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792414384"
Sep 10 12:22:13.976726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:13.976703    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-grlsn\""
Sep 10 12:22:13.979878 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:13.979855    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-140247792414384"]
Sep 10 12:22:14.094861 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:14.094823    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzqtt\" (UniqueName: \"kubernetes.io/projected/a4505b34-5440-4089-bce0-44067a7a7f57-kube-api-access-hzqtt\") pod \"test-curl-7609-140247792414384\" (UID: \"a4505b34-5440-4089-bce0-44067a7a7f57\") " pod="opendatahub/test-curl-7609-140247792414384"
Sep 10 12:22:14.195304 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:14.195271    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzqtt\" (UniqueName: \"kubernetes.io/projected/a4505b34-5440-4089-bce0-44067a7a7f57-kube-api-access-hzqtt\") pod \"test-curl-7609-140247792414384\" (UID: \"a4505b34-5440-4089-bce0-44067a7a7f57\") " pod="opendatahub/test-curl-7609-140247792414384"
Sep 10 12:22:14.205085 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:14.205058    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzqtt\" (UniqueName: \"kubernetes.io/projected/a4505b34-5440-4089-bce0-44067a7a7f57-kube-api-access-hzqtt\") pod \"test-curl-7609-140247792414384\" (UID: \"a4505b34-5440-4089-bce0-44067a7a7f57\") " pod="opendatahub/test-curl-7609-140247792414384"
Sep 10 12:22:14.282165 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:14.282084    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792414384"
Sep 10 12:22:14.552245 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:14.552221    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-140247792414384"]
Sep 10 12:22:14.553783 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:14.553756    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda4505b34_5440_4089_bce0_44067a7a7f57.slice/crio-e295097999b8a4d088580752efe85677b688f100f6330b388f2a79e0ce133887 WatchSource:0}: Error finding container e295097999b8a4d088580752efe85677b688f100f6330b388f2a79e0ce133887: Status 404 returned error can't find the container with id e295097999b8a4d088580752efe85677b688f100f6330b388f2a79e0ce133887
Sep 10 12:22:15.277928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:15.277888    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-140247792414384" event={"ID":"a4505b34-5440-4089-bce0-44067a7a7f57","Type":"ContainerStarted","Data":"e295097999b8a4d088580752efe85677b688f100f6330b388f2a79e0ce133887"}
Sep 10 12:22:15.839963 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:15.839926    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:22:17.288800 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:17.288715    2552 generic.go:356] "Generic (PLEG): container finished" podID="a4505b34-5440-4089-bce0-44067a7a7f57" containerID="ed487f64918380016f057c044245c616e35591d7c4c82ffa29a5b0e4d8a76d3f" exitCode=0
Sep 10 12:22:17.288800 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:17.288787    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-140247792414384" event={"ID":"a4505b34-5440-4089-bce0-44067a7a7f57","Type":"ContainerDied","Data":"ed487f64918380016f057c044245c616e35591d7c4c82ffa29a5b0e4d8a76d3f"}
Sep 10 12:22:18.291404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.291369    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"]
Sep 10 12:22:18.295178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.295156    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.301582 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.301559    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-7576a1\""
Sep 10 12:22:18.308252 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.308229    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"]
Sep 10 12:22:18.421784 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.421762    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792414384"
Sep 10 12:22:18.432724 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.432701    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.432860 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.432770    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpdvp\" (UniqueName: \"kubernetes.io/projected/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-kube-api-access-hpdvp\") pod \"payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.533837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.533809    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hzqtt\" (UniqueName: \"kubernetes.io/projected/a4505b34-5440-4089-bce0-44067a7a7f57-kube-api-access-hzqtt\") pod \"a4505b34-5440-4089-bce0-44067a7a7f57\" (UID: \"a4505b34-5440-4089-bce0-44067a7a7f57\") "
Sep 10 12:22:18.534012 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.533972    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpdvp\" (UniqueName: \"kubernetes.io/projected/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-kube-api-access-hpdvp\") pod \"payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.534079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.534022    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.534720 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.534699    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.536243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.536223    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4505b34-5440-4089-bce0-44067a7a7f57-kube-api-access-hzqtt" (OuterVolumeSpecName: "kube-api-access-hzqtt") pod "a4505b34-5440-4089-bce0-44067a7a7f57" (UID: "a4505b34-5440-4089-bce0-44067a7a7f57"). InnerVolumeSpecName "kube-api-access-hzqtt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:18.551653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.551604    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpdvp\" (UniqueName: \"kubernetes.io/projected/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-kube-api-access-hpdvp\") pod \"payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.553277 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.553261    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7609-140247792414384_a4505b34-5440-4089-bce0-44067a7a7f57/test-curl-7609-140247792414384/0.log"
Sep 10 12:22:18.593551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.593524    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-140247792414384"]
Sep 10 12:22:18.598557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.598534    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-140247792414384"]
Sep 10 12:22:18.607422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.607404    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:18.634974 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.634950    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzqtt\" (UniqueName: \"kubernetes.io/projected/a4505b34-5440-4089-bce0-44067a7a7f57-kube-api-access-hzqtt\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:22:18.721646 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.721617    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7609-140247792411184"]
Sep 10 12:22:18.721935 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.721921    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4505b34-5440-4089-bce0-44067a7a7f57" containerName="test-curl-7609-140247792414384"
Sep 10 12:22:18.721980 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.721937    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4505b34-5440-4089-bce0-44067a7a7f57" containerName="test-curl-7609-140247792414384"
Sep 10 12:22:18.722013 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.721996    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4505b34-5440-4089-bce0-44067a7a7f57" containerName="test-curl-7609-140247792414384"
Sep 10 12:22:18.725979 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.725965    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:18.734401 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.734381    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-140247792411184"]
Sep 10 12:22:18.836155 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.836102    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kprmm\" (UniqueName: \"kubernetes.io/projected/23e5c148-7e98-4690-8c22-fa67b7e78d27-kube-api-access-kprmm\") pod \"test-curl-7609-140247792411184\" (UID: \"23e5c148-7e98-4690-8c22-fa67b7e78d27\") " pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:18.885233 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.885210    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"]
Sep 10 12:22:18.888846 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:18.888823    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad35c8a9_bda9_4033_9bff_93e5b6329bf7.slice/crio-a87c47764ee8ddcd6803103dac7607980f2111ee87c9d03f54c1efe1bba9c6bf WatchSource:0}: Error finding container a87c47764ee8ddcd6803103dac7607980f2111ee87c9d03f54c1efe1bba9c6bf: Status 404 returned error can't find the container with id a87c47764ee8ddcd6803103dac7607980f2111ee87c9d03f54c1efe1bba9c6bf
Sep 10 12:22:18.936847 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.936826    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kprmm\" (UniqueName: \"kubernetes.io/projected/23e5c148-7e98-4690-8c22-fa67b7e78d27-kube-api-access-kprmm\") pod \"test-curl-7609-140247792411184\" (UID: \"23e5c148-7e98-4690-8c22-fa67b7e78d27\") " pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:18.949690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:18.949667    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kprmm\" (UniqueName: \"kubernetes.io/projected/23e5c148-7e98-4690-8c22-fa67b7e78d27-kube-api-access-kprmm\") pod \"test-curl-7609-140247792411184\" (UID: \"23e5c148-7e98-4690-8c22-fa67b7e78d27\") " pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:19.036325 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:19.036305    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:19.241480 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:19.241452    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4505b34-5440-4089-bce0-44067a7a7f57" path="/var/lib/kubelet/pods/a4505b34-5440-4089-bce0-44067a7a7f57/volumes"
Sep 10 12:22:19.301700 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:19.299186    2552 scope.go:117] "RemoveContainer" containerID="ed487f64918380016f057c044245c616e35591d7c4c82ffa29a5b0e4d8a76d3f"
Sep 10 12:22:19.301700 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:19.299334    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792414384"
Sep 10 12:22:19.302468 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:19.301805    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd" event={"ID":"ad35c8a9-bda9-4033-9bff-93e5b6329bf7","Type":"ContainerStarted","Data":"a87c47764ee8ddcd6803103dac7607980f2111ee87c9d03f54c1efe1bba9c6bf"}
Sep 10 12:22:19.334879 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:19.334858    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7609-140247792411184"]
Sep 10 12:22:19.337463 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:19.337441    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod23e5c148_7e98_4690_8c22_fa67b7e78d27.slice/crio-aba8f8f15d0e51cea75478ad482c081e94aaedaf5e9468fd89fecebf7b11732e WatchSource:0}: Error finding container aba8f8f15d0e51cea75478ad482c081e94aaedaf5e9468fd89fecebf7b11732e: Status 404 returned error can't find the container with id aba8f8f15d0e51cea75478ad482c081e94aaedaf5e9468fd89fecebf7b11732e
Sep 10 12:22:20.309331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:20.309246    2552 generic.go:356] "Generic (PLEG): container finished" podID="23e5c148-7e98-4690-8c22-fa67b7e78d27" containerID="d217ace8c11b37dc9e7470074f754bea8d1b12a0582f15f5b904274966844897" exitCode=0
Sep 10 12:22:20.309331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:20.309322    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-140247792411184" event={"ID":"23e5c148-7e98-4690-8c22-fa67b7e78d27","Type":"ContainerDied","Data":"d217ace8c11b37dc9e7470074f754bea8d1b12a0582f15f5b904274966844897"}
Sep 10 12:22:20.309795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:20.309353    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7609-140247792411184" event={"ID":"23e5c148-7e98-4690-8c22-fa67b7e78d27","Type":"ContainerStarted","Data":"aba8f8f15d0e51cea75478ad482c081e94aaedaf5e9468fd89fecebf7b11732e"}
Sep 10 12:22:21.315652 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.315615    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd" event={"ID":"ad35c8a9-bda9-4033-9bff-93e5b6329bf7","Type":"ContainerStarted","Data":"8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74"}
Sep 10 12:22:21.316046 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.315808    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:21.336943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.336875    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd" podStartSLOduration=1.4835617779999999 podStartE2EDuration="3.336853516s" totalImagesPullingTime="1.853291738s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:21.336673402 +0000 UTC m=+752.533189103" watchObservedRunningTime="2026-09-10 12:22:21.336853516 +0000 UTC m=+752.533369218"
Sep 10 12:22:21.445330 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.445307    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:21.468004 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.467982    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7609-140247792411184_23e5c148-7e98-4690-8c22-fa67b7e78d27/test-curl-7609-140247792411184/0.log"
Sep 10 12:22:21.491751 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.491726    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7609-140247792411184"]
Sep 10 12:22:21.494268 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.494248    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7609-140247792411184"]
Sep 10 12:22:21.561265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.561240    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kprmm\" (UniqueName: \"kubernetes.io/projected/23e5c148-7e98-4690-8c22-fa67b7e78d27-kube-api-access-kprmm\") pod \"23e5c148-7e98-4690-8c22-fa67b7e78d27\" (UID: \"23e5c148-7e98-4690-8c22-fa67b7e78d27\") "
Sep 10 12:22:21.563473 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.563444    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23e5c148-7e98-4690-8c22-fa67b7e78d27-kube-api-access-kprmm" (OuterVolumeSpecName: "kube-api-access-kprmm") pod "23e5c148-7e98-4690-8c22-fa67b7e78d27" (UID: "23e5c148-7e98-4690-8c22-fa67b7e78d27"). InnerVolumeSpecName "kube-api-access-kprmm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:22:21.662230 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:21.662196    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kprmm\" (UniqueName: \"kubernetes.io/projected/23e5c148-7e98-4690-8c22-fa67b7e78d27-kube-api-access-kprmm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:22:22.321508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:22.321479    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:22.321508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:22.321497    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aba8f8f15d0e51cea75478ad482c081e94aaedaf5e9468fd89fecebf7b11732e"
Sep 10 12:22:22.331247 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:22:22.331217    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7609-140247792411184\" is forbidden: User \"system:node:ip-10-0-129-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-68.ec2.internal' and this object" podUID="23e5c148-7e98-4690-8c22-fa67b7e78d27" pod="opendatahub/test-curl-7609-140247792411184"
Sep 10 12:22:23.243026 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:23.242990    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23e5c148-7e98-4690-8c22-fa67b7e78d27" path="/var/lib/kubelet/pods/23e5c148-7e98-4690-8c22-fa67b7e78d27/volumes"
Sep 10 12:22:26.362391 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.362356    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"]
Sep 10 12:22:26.362755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.362741    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23e5c148-7e98-4690-8c22-fa67b7e78d27" containerName="test-curl-7609-140247792411184"
Sep 10 12:22:26.362795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.362756    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="23e5c148-7e98-4690-8c22-fa67b7e78d27" containerName="test-curl-7609-140247792411184"
Sep 10 12:22:26.362828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.362819    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="23e5c148-7e98-4690-8c22-fa67b7e78d27" containerName="test-curl-7609-140247792411184"
Sep 10 12:22:26.365793 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.365778    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.368566 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.368549    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-772cfbbb-dockercfg-ndb8v\""
Sep 10 12:22:26.368939 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.368923    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-772cfbbb\""
Sep 10 12:22:26.381755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.381732    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"]
Sep 10 12:22:26.411512 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.411478    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"]
Sep 10 12:22:26.415260 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.415239    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.428730 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.428707    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"]
Sep 10 12:22:26.507005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.506985    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hs62\" (UniqueName: \"kubernetes.io/projected/fa4d623b-9816-4c3a-8ca2-d9758546d785-kube-api-access-8hs62\") pod \"payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.507104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.507013    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4d623b-9816-4c3a-8ca2-d9758546d785-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.507104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.507071    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg47p\" (UniqueName: \"kubernetes.io/projected/ea85d09e-5129-491d-9fc4-1314ba793f30-kube-api-access-fg47p\") pod \"payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") " pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.507175 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.507113    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea85d09e-5129-491d-9fc4-1314ba793f30-plugins-config-volume\") pod \"payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") " pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.607806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.607783    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea85d09e-5129-491d-9fc4-1314ba793f30-plugins-config-volume\") pod \"payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") " pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.607899 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.607818    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hs62\" (UniqueName: \"kubernetes.io/projected/fa4d623b-9816-4c3a-8ca2-d9758546d785-kube-api-access-8hs62\") pod \"payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.607964 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.607939    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4d623b-9816-4c3a-8ca2-d9758546d785-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.608084 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.608069    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg47p\" (UniqueName: \"kubernetes.io/projected/ea85d09e-5129-491d-9fc4-1314ba793f30-kube-api-access-fg47p\") pod \"payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") " pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.608696 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.608668    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea85d09e-5129-491d-9fc4-1314ba793f30-plugins-config-volume\") pod \"payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") " pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.608814 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.608674    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4d623b-9816-4c3a-8ca2-d9758546d785-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.620192 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.620175    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hs62\" (UniqueName: \"kubernetes.io/projected/fa4d623b-9816-4c3a-8ca2-d9758546d785-kube-api-access-8hs62\") pod \"payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.620314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.620294    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg47p\" (UniqueName: \"kubernetes.io/projected/ea85d09e-5129-491d-9fc4-1314ba793f30-kube-api-access-fg47p\") pod \"payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") " pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.675480 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.675419    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:26.725273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.725082    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:26.952278 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:26.952198    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"]
Sep 10 12:22:26.954829 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:26.954798    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa4d623b_9816_4c3a_8ca2_d9758546d785.slice/crio-c976019bdbd61a68db54d845111111b3f33e36967272767c28433087d4354739 WatchSource:0}: Error finding container c976019bdbd61a68db54d845111111b3f33e36967272767c28433087d4354739: Status 404 returned error can't find the container with id c976019bdbd61a68db54d845111111b3f33e36967272767c28433087d4354739
Sep 10 12:22:27.021463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.021419    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"]
Sep 10 12:22:27.024304 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:27.024277    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea85d09e_5129_491d_9fc4_1314ba793f30.slice/crio-0c6773adef464a2427840ee0ad74bcc152ff0929c341b75c28e3ef996448f692 WatchSource:0}: Error finding container 0c6773adef464a2427840ee0ad74bcc152ff0929c341b75c28e3ef996448f692: Status 404 returned error can't find the container with id 0c6773adef464a2427840ee0ad74bcc152ff0929c341b75c28e3ef996448f692
Sep 10 12:22:27.346895 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.346810    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w" event={"ID":"ea85d09e-5129-491d-9fc4-1314ba793f30","Type":"ContainerStarted","Data":"404794c1504402aabe59f10aac11759d03118e2ebcdead69c1bffd6c1afc156a"}
Sep 10 12:22:27.347147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.347121    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:27.347261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.347169    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w" event={"ID":"ea85d09e-5129-491d-9fc4-1314ba793f30","Type":"ContainerStarted","Data":"0c6773adef464a2427840ee0ad74bcc152ff0929c341b75c28e3ef996448f692"}
Sep 10 12:22:27.348605 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.348583    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls" event={"ID":"fa4d623b-9816-4c3a-8ca2-d9758546d785","Type":"ContainerStarted","Data":"a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37"}
Sep 10 12:22:27.348709 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.348625    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls" event={"ID":"fa4d623b-9816-4c3a-8ca2-d9758546d785","Type":"ContainerStarted","Data":"c976019bdbd61a68db54d845111111b3f33e36967272767c28433087d4354739"}
Sep 10 12:22:27.348709 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.348656    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:27.371716 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.371677    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w" podStartSLOduration=1.371665349 podStartE2EDuration="1.371665349s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:27.368809248 +0000 UTC m=+758.565324949" watchObservedRunningTime="2026-09-10 12:22:27.371665349 +0000 UTC m=+758.568181150"
Sep 10 12:22:27.389292 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:27.389252    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls" podStartSLOduration=1.389238699 podStartE2EDuration="1.389238699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:27.388515752 +0000 UTC m=+758.585031455" watchObservedRunningTime="2026-09-10 12:22:27.389238699 +0000 UTC m=+758.585754399"
Sep 10 12:22:28.398847 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.398818    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"]
Sep 10 12:22:28.402143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.402126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.406191 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.406167    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-98aa83-dockercfg-dc22b\""
Sep 10 12:22:28.406285 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.406199    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-98aa83\""
Sep 10 12:22:28.419277 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.419253    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"]
Sep 10 12:22:28.430789 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.430762    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"]
Sep 10 12:22:28.434568 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.434550    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.446954 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.446935    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"]
Sep 10 12:22:28.525748 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.525718    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.525854 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.525768    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-plugins-config-volume\") pod \"payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") " pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.525854 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.525795    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvjrz\" (UniqueName: \"kubernetes.io/projected/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-kube-api-access-wvjrz\") pod \"payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") " pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.525854 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.525819    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vsx2\" (UniqueName: \"kubernetes.io/projected/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-kube-api-access-4vsx2\") pod \"payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.626810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.626785    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.626926 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.626828    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-plugins-config-volume\") pod \"payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") " pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.626926 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.626856    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvjrz\" (UniqueName: \"kubernetes.io/projected/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-kube-api-access-wvjrz\") pod \"payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") " pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.626926 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.626879    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vsx2\" (UniqueName: \"kubernetes.io/projected/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-kube-api-access-4vsx2\") pod \"payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.627535 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.627518    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-plugins-config-volume\") pod \"payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") " pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.627607 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.627555    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.636370 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.636349    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vsx2\" (UniqueName: \"kubernetes.io/projected/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-kube-api-access-4vsx2\") pod \"payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.636802 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.636783    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvjrz\" (UniqueName: \"kubernetes.io/projected/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-kube-api-access-wvjrz\") pod \"payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") " pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.712769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.712714    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:28.744002 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.743967    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:28.989071 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:28.989045    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"]
Sep 10 12:22:28.991410 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:28.991385    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd71ef12f_063c_4bd8_b5b1_6dc0262ea2d4.slice/crio-f5f9279f704f10fc9d4fb2ddb37d0004bb14209701adc1974dc3212f994e202a WatchSource:0}: Error finding container f5f9279f704f10fc9d4fb2ddb37d0004bb14209701adc1974dc3212f994e202a: Status 404 returned error can't find the container with id f5f9279f704f10fc9d4fb2ddb37d0004bb14209701adc1974dc3212f994e202a
Sep 10 12:22:29.033124 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.033095    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"]
Sep 10 12:22:29.033818 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:29.033794    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70a45bc5_aee1_4fd0_9cd4_cfa5adaf26d6.slice/crio-2b3c24658de5e7f651ede04c31008923d9352399cd57e5bc1da3ac1d8783a438 WatchSource:0}: Error finding container 2b3c24658de5e7f651ede04c31008923d9352399cd57e5bc1da3ac1d8783a438: Status 404 returned error can't find the container with id 2b3c24658de5e7f651ede04c31008923d9352399cd57e5bc1da3ac1d8783a438
Sep 10 12:22:29.358248 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.358160    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb" event={"ID":"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4","Type":"ContainerStarted","Data":"171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd"}
Sep 10 12:22:29.358248 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.358209    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb" event={"ID":"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4","Type":"ContainerStarted","Data":"f5f9279f704f10fc9d4fb2ddb37d0004bb14209701adc1974dc3212f994e202a"}
Sep 10 12:22:29.358248 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.358236    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:29.359728 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.359704    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n" event={"ID":"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6","Type":"ContainerStarted","Data":"52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9"}
Sep 10 12:22:29.359855 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.359746    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n" event={"ID":"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6","Type":"ContainerStarted","Data":"2b3c24658de5e7f651ede04c31008923d9352399cd57e5bc1da3ac1d8783a438"}
Sep 10 12:22:29.359855 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.359774    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:29.549711 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.549656    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb" podStartSLOduration=1.5496405709999999 podStartE2EDuration="1.549640571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:29.548121741 +0000 UTC m=+760.744637444" watchObservedRunningTime="2026-09-10 12:22:29.549640571 +0000 UTC m=+760.746156270"
Sep 10 12:22:29.586295 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.586254    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n" podStartSLOduration=1.58623917 podStartE2EDuration="1.58623917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:29.585859449 +0000 UTC m=+760.782375149" watchObservedRunningTime="2026-09-10 12:22:29.58623917 +0000 UTC m=+760.782754871"
Sep 10 12:22:29.697021 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:29.696990    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:22:32.323941 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:32.323913    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:22:36.255127 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.255095    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"]
Sep 10 12:22:36.261880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.261860    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.269994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.269967    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-be65aff6-kserve-self-signed-certs\""
Sep 10 12:22:36.273060 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.273037    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"]
Sep 10 12:22:36.394930 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.394901    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46936de5-52a4-4733-9634-879f7fd35dfc-tls-certs\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.395121 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.394952    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-model-cache\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.395121 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.394972    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-home\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.395121 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.395068    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-dshm\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.395299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.395196    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2xtd\" (UniqueName: \"kubernetes.io/projected/46936de5-52a4-4733-9634-879f7fd35dfc-kube-api-access-b2xtd\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.395299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.395232    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-tmp-dir\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.496735 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.496702    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2xtd\" (UniqueName: \"kubernetes.io/projected/46936de5-52a4-4733-9634-879f7fd35dfc-kube-api-access-b2xtd\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.496735 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.496743    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-tmp-dir\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.496991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.496765    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46936de5-52a4-4733-9634-879f7fd35dfc-tls-certs\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.496991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.496798    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-model-cache\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.496991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.496816    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-home\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.496991 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.496836    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-dshm\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.497183 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.497154    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-tmp-dir\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.497260 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.497229    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-model-cache\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.497563 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.497504    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-home\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.499176 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.499149    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-dshm\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.499511 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.499489    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46936de5-52a4-4733-9634-879f7fd35dfc-tls-certs\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.505975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.505914    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2xtd\" (UniqueName: \"kubernetes.io/projected/46936de5-52a4-4733-9634-879f7fd35dfc-kube-api-access-b2xtd\") pod \"e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") " pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.570993 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.570952    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:36.846124 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:36.846100    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"]
Sep 10 12:22:36.848720 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:36.848688    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod46936de5_52a4_4733_9634_879f7fd35dfc.slice/crio-411550a761503e3131f0731c85f8b5231aac132b66e9ffa7023f6fc63cd2b29e WatchSource:0}: Error finding container 411550a761503e3131f0731c85f8b5231aac132b66e9ffa7023f6fc63cd2b29e: Status 404 returned error can't find the container with id 411550a761503e3131f0731c85f8b5231aac132b66e9ffa7023f6fc63cd2b29e
Sep 10 12:22:37.398452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:37.398378    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f" event={"ID":"46936de5-52a4-4733-9634-879f7fd35dfc","Type":"ContainerStarted","Data":"023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36"}
Sep 10 12:22:37.398452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:37.398421    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f" event={"ID":"46936de5-52a4-4733-9634-879f7fd35dfc","Type":"ContainerStarted","Data":"411550a761503e3131f0731c85f8b5231aac132b66e9ffa7023f6fc63cd2b29e"}
Sep 10 12:22:37.423224 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:37.423168    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f" podStartSLOduration=1.423151899 podStartE2EDuration="1.423151899s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:37.421528632 +0000 UTC m=+768.618044331" watchObservedRunningTime="2026-09-10 12:22:37.423151899 +0000 UTC m=+768.619667618"
Sep 10 12:22:38.355224 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:38.355192    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:22:38.355391 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:38.355263    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:22:40.366627 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:40.366581    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:22:40.367005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:40.366763    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:22:45.272208 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:45.272172    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:22:46.571881 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:46.571845    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:46.572272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:46.571896    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:46.584507 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:46.584480    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:47.463034 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.463006    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:22:47.589685 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.589656    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"]
Sep 10 12:22:47.593051 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.593034    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:47.597209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.597189    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fb35b9\""
Sep 10 12:22:47.597694 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.597673    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fb35b9-dockercfg-hctfn\""
Sep 10 12:22:47.613915 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.613894    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"]
Sep 10 12:22:47.697001 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.696962    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9bpf\" (UniqueName: \"kubernetes.io/projected/80703ce5-2296-4b40-95bd-ad104c73189c-kube-api-access-f9bpf\") pod \"payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:47.697164 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.697020    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80703ce5-2296-4b40-95bd-ad104c73189c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:47.798463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.798353    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9bpf\" (UniqueName: \"kubernetes.io/projected/80703ce5-2296-4b40-95bd-ad104c73189c-kube-api-access-f9bpf\") pod \"payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:47.798463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.798396    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80703ce5-2296-4b40-95bd-ad104c73189c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:47.799116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.799093    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80703ce5-2296-4b40-95bd-ad104c73189c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:47.809318 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.809295    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9bpf\" (UniqueName: \"kubernetes.io/projected/80703ce5-2296-4b40-95bd-ad104c73189c-kube-api-access-f9bpf\") pod \"payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:47.904481 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:47.904420    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:48.184329 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:48.184305    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"]
Sep 10 12:22:48.186305 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:48.186275    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80703ce5_2296_4b40_95bd_ad104c73189c.slice/crio-3fb304bd164bc082d12c77d9610fc6a5208cbed2db179d460b31e75efbc72803 WatchSource:0}: Error finding container 3fb304bd164bc082d12c77d9610fc6a5208cbed2db179d460b31e75efbc72803: Status 404 returned error can't find the container with id 3fb304bd164bc082d12c77d9610fc6a5208cbed2db179d460b31e75efbc72803
Sep 10 12:22:48.457303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:48.457211    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn" event={"ID":"80703ce5-2296-4b40-95bd-ad104c73189c","Type":"ContainerStarted","Data":"529a1bbbc59a8bf241f84d9ebe7e81e5892abc98f339cbf715efbc72d6f6f5e2"}
Sep 10 12:22:48.457303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:48.457273    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn" event={"ID":"80703ce5-2296-4b40-95bd-ad104c73189c","Type":"ContainerStarted","Data":"3fb304bd164bc082d12c77d9610fc6a5208cbed2db179d460b31e75efbc72803"}
Sep 10 12:22:48.457588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:48.457572    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:22:48.479024 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:48.478973    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn" podStartSLOduration=1.478958108 podStartE2EDuration="1.478958108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:48.477157179 +0000 UTC m=+779.673672879" watchObservedRunningTime="2026-09-10 12:22:48.478958108 +0000 UTC m=+779.675473799"
Sep 10 12:22:49.805352 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:49.805318    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"]
Sep 10 12:22:49.810083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:49.810065    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"
Sep 10 12:22:49.813443 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:49.813407    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:22:49.813443 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:49.813413    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:22:49.820807 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:49.820783    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"]
Sep 10 12:22:49.916157 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:49.916124    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9ca183-2273-446f-8265-c7b6ea1ef7c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5\" (UID: \"be9ca183-2273-446f-8265-c7b6ea1ef7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"
Sep 10 12:22:50.017553 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:50.017516    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9ca183-2273-446f-8265-c7b6ea1ef7c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5\" (UID: \"be9ca183-2273-446f-8265-c7b6ea1ef7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"
Sep 10 12:22:50.018106 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:50.018088    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9ca183-2273-446f-8265-c7b6ea1ef7c6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5\" (UID: \"be9ca183-2273-446f-8265-c7b6ea1ef7c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"
Sep 10 12:22:50.120513 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:50.120486    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"
Sep 10 12:22:50.402888 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:50.402867    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"]
Sep 10 12:22:50.403459 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:22:50.403411    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe9ca183_2273_446f_8265_c7b6ea1ef7c6.slice/crio-0ca419af2e171c2b6296f48edc33b295470bd2785c2752b433fcf045cece6fbd WatchSource:0}: Error finding container 0ca419af2e171c2b6296f48edc33b295470bd2785c2752b433fcf045cece6fbd: Status 404 returned error can't find the container with id 0ca419af2e171c2b6296f48edc33b295470bd2785c2752b433fcf045cece6fbd
Sep 10 12:22:50.467990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:50.467963    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5" event={"ID":"be9ca183-2273-446f-8265-c7b6ea1ef7c6","Type":"ContainerStarted","Data":"0ca419af2e171c2b6296f48edc33b295470bd2785c2752b433fcf045cece6fbd"}
Sep 10 12:22:53.490762 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:53.490710    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5" event={"ID":"be9ca183-2273-446f-8265-c7b6ea1ef7c6","Type":"ContainerStarted","Data":"b992249d96ade00e6a0bd3811be05846d73059cd47b8001a77a266c74ffdad93"}
Sep 10 12:22:53.539851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:53.539786    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5" podStartSLOduration=2.172939341 podStartE2EDuration="4.539766403s" totalImagesPullingTime="2.366827062s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:22:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:22:53.539767423 +0000 UTC m=+784.736283122" watchObservedRunningTime="2026-09-10 12:22:53.539766403 +0000 UTC m=+784.736282103"
Sep 10 12:22:54.497151 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:54.497111    2552 generic.go:356] "Generic (PLEG): container finished" podID="be9ca183-2273-446f-8265-c7b6ea1ef7c6" containerID="b992249d96ade00e6a0bd3811be05846d73059cd47b8001a77a266c74ffdad93" exitCode=0
Sep 10 12:22:54.497652 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:54.497197    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5" event={"ID":"be9ca183-2273-446f-8265-c7b6ea1ef7c6","Type":"ContainerDied","Data":"b992249d96ade00e6a0bd3811be05846d73059cd47b8001a77a266c74ffdad93"}
Sep 10 12:22:55.632751 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:55.632726    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"
Sep 10 12:22:55.669803 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:55.669780    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9ca183-2273-446f-8265-c7b6ea1ef7c6-maas-api-service-ca\") pod \"be9ca183-2273-446f-8265-c7b6ea1ef7c6\" (UID: \"be9ca183-2273-446f-8265-c7b6ea1ef7c6\") "
Sep 10 12:22:55.670118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:55.670098    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be9ca183-2273-446f-8265-c7b6ea1ef7c6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "be9ca183-2273-446f-8265-c7b6ea1ef7c6" (UID: "be9ca183-2273-446f-8265-c7b6ea1ef7c6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:22:55.770476 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:55.770409    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/be9ca183-2273-446f-8265-c7b6ea1ef7c6-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:22:56.508333 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:56.508300    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"
Sep 10 12:22:56.508519 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:56.508340    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5" event={"ID":"be9ca183-2273-446f-8265-c7b6ea1ef7c6","Type":"ContainerDied","Data":"0ca419af2e171c2b6296f48edc33b295470bd2785c2752b433fcf045cece6fbd"}
Sep 10 12:22:56.508519 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:56.508394    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ca419af2e171c2b6296f48edc33b295470bd2785c2752b433fcf045cece6fbd"
Sep 10 12:22:58.554804 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:22:58.554774    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:23:00.470759 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:00.470729    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:23:06.436025 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.435994    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"]
Sep 10 12:23:06.436490 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.436246    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w" podUID="ea85d09e-5129-491d-9fc4-1314ba793f30" containerName="payload-processing" containerID="cri-o://404794c1504402aabe59f10aac11759d03118e2ebcdead69c1bffd6c1afc156a" gracePeriod=30
Sep 10 12:23:06.449180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.449100    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"]
Sep 10 12:23:06.449358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.449312    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls" podUID="fa4d623b-9816-4c3a-8ca2-d9758546d785" containerName="payload-pre-processing" containerID="cri-o://a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37" gracePeriod=30
Sep 10 12:23:06.554232 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.554203    2552 generic.go:356] "Generic (PLEG): container finished" podID="ea85d09e-5129-491d-9fc4-1314ba793f30" containerID="404794c1504402aabe59f10aac11759d03118e2ebcdead69c1bffd6c1afc156a" exitCode=0
Sep 10 12:23:06.554384 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.554280    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w" event={"ID":"ea85d09e-5129-491d-9fc4-1314ba793f30","Type":"ContainerDied","Data":"404794c1504402aabe59f10aac11759d03118e2ebcdead69c1bffd6c1afc156a"}
Sep 10 12:23:06.635959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.635929    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"]
Sep 10 12:23:06.641052 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.641028    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-772cfbbb-qksn5"]
Sep 10 12:23:06.722222 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.722202    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:23:06.724944 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.724924    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:23:06.874593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.874559    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8hs62\" (UniqueName: \"kubernetes.io/projected/fa4d623b-9816-4c3a-8ca2-d9758546d785-kube-api-access-8hs62\") pod \"fa4d623b-9816-4c3a-8ca2-d9758546d785\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") "
Sep 10 12:23:06.874773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.874638    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fg47p\" (UniqueName: \"kubernetes.io/projected/ea85d09e-5129-491d-9fc4-1314ba793f30-kube-api-access-fg47p\") pod \"ea85d09e-5129-491d-9fc4-1314ba793f30\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") "
Sep 10 12:23:06.874773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.874658    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4d623b-9816-4c3a-8ca2-d9758546d785-plugins-config-volume\") pod \"fa4d623b-9816-4c3a-8ca2-d9758546d785\" (UID: \"fa4d623b-9816-4c3a-8ca2-d9758546d785\") "
Sep 10 12:23:06.874773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.874719    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea85d09e-5129-491d-9fc4-1314ba793f30-plugins-config-volume\") pod \"ea85d09e-5129-491d-9fc4-1314ba793f30\" (UID: \"ea85d09e-5129-491d-9fc4-1314ba793f30\") "
Sep 10 12:23:06.875048 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.875017    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa4d623b-9816-4c3a-8ca2-d9758546d785-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa4d623b-9816-4c3a-8ca2-d9758546d785" (UID: "fa4d623b-9816-4c3a-8ca2-d9758546d785"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:23:06.875136 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.875107    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea85d09e-5129-491d-9fc4-1314ba793f30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ea85d09e-5129-491d-9fc4-1314ba793f30" (UID: "ea85d09e-5129-491d-9fc4-1314ba793f30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:23:06.876969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.876949    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa4d623b-9816-4c3a-8ca2-d9758546d785-kube-api-access-8hs62" (OuterVolumeSpecName: "kube-api-access-8hs62") pod "fa4d623b-9816-4c3a-8ca2-d9758546d785" (UID: "fa4d623b-9816-4c3a-8ca2-d9758546d785"). InnerVolumeSpecName "kube-api-access-8hs62". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:23:06.877038 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.876970    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea85d09e-5129-491d-9fc4-1314ba793f30-kube-api-access-fg47p" (OuterVolumeSpecName: "kube-api-access-fg47p") pod "ea85d09e-5129-491d-9fc4-1314ba793f30" (UID: "ea85d09e-5129-491d-9fc4-1314ba793f30"). InnerVolumeSpecName "kube-api-access-fg47p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:23:06.976260 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.976179    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg47p\" (UniqueName: \"kubernetes.io/projected/ea85d09e-5129-491d-9fc4-1314ba793f30-kube-api-access-fg47p\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:23:06.976260 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.976209    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa4d623b-9816-4c3a-8ca2-d9758546d785-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:23:06.976260 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.976219    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ea85d09e-5129-491d-9fc4-1314ba793f30-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:23:06.976260 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:06.976227    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hs62\" (UniqueName: \"kubernetes.io/projected/fa4d623b-9816-4c3a-8ca2-d9758546d785-kube-api-access-8hs62\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:23:07.242537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.242459    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="be9ca183-2273-446f-8265-c7b6ea1ef7c6" path="/var/lib/kubelet/pods/be9ca183-2273-446f-8265-c7b6ea1ef7c6/volumes"
Sep 10 12:23:07.561534 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.561452    2552 generic.go:356] "Generic (PLEG): container finished" podID="fa4d623b-9816-4c3a-8ca2-d9758546d785" containerID="a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37" exitCode=0
Sep 10 12:23:07.561534 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.561491    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"
Sep 10 12:23:07.561994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.561536    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls" event={"ID":"fa4d623b-9816-4c3a-8ca2-d9758546d785","Type":"ContainerDied","Data":"a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37"}
Sep 10 12:23:07.561994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.561590    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls" event={"ID":"fa4d623b-9816-4c3a-8ca2-d9758546d785","Type":"ContainerDied","Data":"c976019bdbd61a68db54d845111111b3f33e36967272767c28433087d4354739"}
Sep 10 12:23:07.561994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.561620    2552 scope.go:117] "RemoveContainer" containerID="a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37"
Sep 10 12:23:07.563072 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.563058    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"
Sep 10 12:23:07.563072 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.563060    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w" event={"ID":"ea85d09e-5129-491d-9fc4-1314ba793f30","Type":"ContainerDied","Data":"0c6773adef464a2427840ee0ad74bcc152ff0929c341b75c28e3ef996448f692"}
Sep 10 12:23:07.570439 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.570407    2552 scope.go:117] "RemoveContainer" containerID="a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37"
Sep 10 12:23:07.570729 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:23:07.570709    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37\": container with ID starting with a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37 not found: ID does not exist" containerID="a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37"
Sep 10 12:23:07.570802 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.570736    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37"} err="failed to get container status \"a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37\": rpc error: code = NotFound desc = could not find container \"a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37\": container with ID starting with a66a2486a7931047bba9990ba705c652e837f6d1ba1a556af219e299d856fd37 not found: ID does not exist"
Sep 10 12:23:07.570802 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.570752    2552 scope.go:117] "RemoveContainer" containerID="404794c1504402aabe59f10aac11759d03118e2ebcdead69c1bffd6c1afc156a"
Sep 10 12:23:07.581732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.581713    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"]
Sep 10 12:23:07.587742 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.587723    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-772cfbbb-744ccf7f65-th79w"]
Sep 10 12:23:07.601002 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.600980    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"]
Sep 10 12:23:07.603635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.603613    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-772cfbbb-6f57569b58-6gvls"]
Sep 10 12:23:07.761449 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761405    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"]
Sep 10 12:23:07.761799 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761786    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa4d623b-9816-4c3a-8ca2-d9758546d785" containerName="payload-pre-processing"
Sep 10 12:23:07.761850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761800    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa4d623b-9816-4c3a-8ca2-d9758546d785" containerName="payload-pre-processing"
Sep 10 12:23:07.761850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761808    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea85d09e-5129-491d-9fc4-1314ba793f30" containerName="payload-processing"
Sep 10 12:23:07.761850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761814    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea85d09e-5129-491d-9fc4-1314ba793f30" containerName="payload-processing"
Sep 10 12:23:07.761850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761823    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="be9ca183-2273-446f-8265-c7b6ea1ef7c6" containerName="revoke-keys"
Sep 10 12:23:07.761850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761829    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="be9ca183-2273-446f-8265-c7b6ea1ef7c6" containerName="revoke-keys"
Sep 10 12:23:07.762010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761876    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="be9ca183-2273-446f-8265-c7b6ea1ef7c6" containerName="revoke-keys"
Sep 10 12:23:07.762010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761883    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa4d623b-9816-4c3a-8ca2-d9758546d785" containerName="payload-pre-processing"
Sep 10 12:23:07.762010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.761890    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea85d09e-5129-491d-9fc4-1314ba793f30" containerName="payload-processing"
Sep 10 12:23:07.766243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.766225    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.768844 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.768821    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-41f030dc-gw-openshift-default-dockercfg-snpzw\""
Sep 10 12:23:07.768969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.768862    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 12:23:07.788803 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.788778    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"]
Sep 10 12:23:07.883582 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883555    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istiod-ca-cert\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.883744 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883599    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-envoy\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.883744 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883687    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-podinfo\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.883744 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883724    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4ssqk\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-kube-api-access-4ssqk\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.883866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883747    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-token\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.883866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883811    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-credential-socket\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.883866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883832    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-certs\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.883968 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883892    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-data\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.884007 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.883962    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-socket\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985065    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4ssqk\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-kube-api-access-4ssqk\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985104    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-token\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985141    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-credential-socket\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985161    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-certs\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985183    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-data\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985232    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-socket\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985296    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istiod-ca-cert\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985322    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-envoy\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985686 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985370    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-podinfo\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985686 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985635    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-certs\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985709    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-credential-socket\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985734    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-socket\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.985889 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.985833    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-data\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.986042 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.986014    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istiod-ca-cert\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.987814 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.987786    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-envoy\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.988040 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.988022    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-podinfo\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:07.999274 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:07.999251    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-token\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:08.003319 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:08.003286    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4ssqk\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-kube-api-access-4ssqk\") pod \"e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") " pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:08.076404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:08.076376    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:08.368707 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:08.368682    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"]
Sep 10 12:23:08.370386 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:23:08.370359    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a0d53e_f631_4c16_a5ff_f29c5b7a6e78.slice/crio-e1802fc2622848ff6f33d22deb665e21b86b43b2eb71d7cacb9849853f5945cf WatchSource:0}: Error finding container e1802fc2622848ff6f33d22deb665e21b86b43b2eb71d7cacb9849853f5945cf: Status 404 returned error can't find the container with id e1802fc2622848ff6f33d22deb665e21b86b43b2eb71d7cacb9849853f5945cf
Sep 10 12:23:08.569962 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:08.569921    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" event={"ID":"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78","Type":"ContainerStarted","Data":"e1802fc2622848ff6f33d22deb665e21b86b43b2eb71d7cacb9849853f5945cf"}
Sep 10 12:23:09.242266 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:09.242237    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea85d09e-5129-491d-9fc4-1314ba793f30" path="/var/lib/kubelet/pods/ea85d09e-5129-491d-9fc4-1314ba793f30/volumes"
Sep 10 12:23:09.242590 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:09.242575    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa4d623b-9816-4c3a-8ca2-d9758546d785" path="/var/lib/kubelet/pods/fa4d623b-9816-4c3a-8ca2-d9758546d785/volumes"
Sep 10 12:23:12.048638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:12.048550    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:23:12.373123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:12.373078    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:23:12.373254 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:12.373173    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:23:12.373254 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:12.373219    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:23:12.591794 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:12.591749    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" event={"ID":"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78","Type":"ContainerStarted","Data":"7f4e04176ae407270e6138d8857151438323e1590f7d8813316fc27585782b2c"}
Sep 10 12:23:12.618265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:12.618206    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podStartSLOduration=1.617802848 podStartE2EDuration="5.618183384s" totalImagesPullingTime="4.000380536s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:23:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:23:12.6151903 +0000 UTC m=+803.811706002" watchObservedRunningTime="2026-09-10 12:23:12.618183384 +0000 UTC m=+803.814699086"
Sep 10 12:23:13.077091 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.077056    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:13.078814 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.078785    2552 patch_prober.go:28] interesting pod/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 12:23:13.078928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.078847    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 12:23:13.346950 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.346858    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"]
Sep 10 12:23:13.350393 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.350371    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.353458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.353438    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-be65aff6-kserve-self-signed-certs\""
Sep 10 12:23:13.361665 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.361647    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"]
Sep 10 12:23:13.436170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.436135    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4npz\" (UniqueName: \"kubernetes.io/projected/908d8713-d558-4ba5-a708-3f1a8c36c39f-kube-api-access-t4npz\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.436170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.436181    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d8713-d558-4ba5-a708-3f1a8c36c39f-tls-certs\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.436368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.436200    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-home\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.436368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.436269    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-dshm\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.436368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.436318    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-model-cache\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.436368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.436347    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-tmp-dir\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537038 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537010    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-model-cache\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537205 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537055    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-tmp-dir\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537205 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537080    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4npz\" (UniqueName: \"kubernetes.io/projected/908d8713-d558-4ba5-a708-3f1a8c36c39f-kube-api-access-t4npz\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537205 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537106    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d8713-d558-4ba5-a708-3f1a8c36c39f-tls-certs\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537205 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537124    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-home\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537265    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-dshm\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537527 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537495    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-model-cache\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537555    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-tmp-dir\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.537630 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.537589    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-home\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.539469 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.539446    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-dshm\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.539588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.539555    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d8713-d558-4ba5-a708-3f1a8c36c39f-tls-certs\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.546450 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.546404    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4npz\" (UniqueName: \"kubernetes.io/projected/908d8713-d558-4ba5-a708-3f1a8c36c39f-kube-api-access-t4npz\") pod \"e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") " pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.659751 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.659706    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:13.934903 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:13.934869    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"]
Sep 10 12:23:13.936941 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:23:13.936914    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod908d8713_d558_4ba5_a708_3f1a8c36c39f.slice/crio-0375c08cf07eaccaaa99c4cba0b88570c33d49c34e88ff9ad3986b14909e9013 WatchSource:0}: Error finding container 0375c08cf07eaccaaa99c4cba0b88570c33d49c34e88ff9ad3986b14909e9013: Status 404 returned error can't find the container with id 0375c08cf07eaccaaa99c4cba0b88570c33d49c34e88ff9ad3986b14909e9013
Sep 10 12:23:14.080624 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:14.080596    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:14.603272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:14.603224    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7" event={"ID":"908d8713-d558-4ba5-a708-3f1a8c36c39f","Type":"ContainerStarted","Data":"5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d"}
Sep 10 12:23:14.603272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:14.603276    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7" event={"ID":"908d8713-d558-4ba5-a708-3f1a8c36c39f","Type":"ContainerStarted","Data":"0375c08cf07eaccaaa99c4cba0b88570c33d49c34e88ff9ad3986b14909e9013"}
Sep 10 12:23:14.603521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:14.603476    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:14.604350 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:14.604325    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:14.626260 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:14.626214    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7" podStartSLOduration=1.626199337 podStartE2EDuration="1.626199337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:23:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:23:14.625662749 +0000 UTC m=+805.822178449" watchObservedRunningTime="2026-09-10 12:23:14.626199337 +0000 UTC m=+805.822715037"
Sep 10 12:23:15.263915 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.263886    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"]
Sep 10 12:23:15.266913 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.266898    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.273778 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.273754    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-41f030dc\""
Sep 10 12:23:15.274209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.273962    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-41f030dc-dockercfg-nnf8d\""
Sep 10 12:23:15.291600 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.291578    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"]
Sep 10 12:23:15.355014 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.354992    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760f0b88-0d50-42e0-932e-011ac99923a3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") " pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.355125 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.355032    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fv47c\" (UniqueName: \"kubernetes.io/projected/760f0b88-0d50-42e0-932e-011ac99923a3-kube-api-access-fv47c\") pod \"payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") " pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.456263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.456235    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760f0b88-0d50-42e0-932e-011ac99923a3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") " pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.456367 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.456287    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fv47c\" (UniqueName: \"kubernetes.io/projected/760f0b88-0d50-42e0-932e-011ac99923a3-kube-api-access-fv47c\") pod \"payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") " pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.456909 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.456892    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760f0b88-0d50-42e0-932e-011ac99923a3-plugins-config-volume\") pod \"payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") " pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.465460 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.465423    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fv47c\" (UniqueName: \"kubernetes.io/projected/760f0b88-0d50-42e0-932e-011ac99923a3-kube-api-access-fv47c\") pod \"payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") " pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.578232 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.578174    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:15.856071 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:15.856045    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"]
Sep 10 12:23:15.857766 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:23:15.857742    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod760f0b88_0d50_42e0_932e_011ac99923a3.slice/crio-868b6f3c621248112d4f542b7e89b1fba06475f10edaa8dc33ff91b9d25be67f WatchSource:0}: Error finding container 868b6f3c621248112d4f542b7e89b1fba06475f10edaa8dc33ff91b9d25be67f: Status 404 returned error can't find the container with id 868b6f3c621248112d4f542b7e89b1fba06475f10edaa8dc33ff91b9d25be67f
Sep 10 12:23:16.615870 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:16.615828    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg" event={"ID":"760f0b88-0d50-42e0-932e-011ac99923a3","Type":"ContainerStarted","Data":"26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0"}
Sep 10 12:23:16.616444 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:16.615932    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg" event={"ID":"760f0b88-0d50-42e0-932e-011ac99923a3","Type":"ContainerStarted","Data":"868b6f3c621248112d4f542b7e89b1fba06475f10edaa8dc33ff91b9d25be67f"}
Sep 10 12:23:16.639381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:16.639336    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg" podStartSLOduration=1.639316441 podStartE2EDuration="1.639316441s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:23:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:23:16.637350655 +0000 UTC m=+807.833866371" watchObservedRunningTime="2026-09-10 12:23:16.639316441 +0000 UTC m=+807.835832140"
Sep 10 12:23:17.621510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:17.621482    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:23.659943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:23.659892    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:23.659943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:23.659948    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:23.672550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:23.672524    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:24.664736 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:24.664708    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:23:28.628110 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:28.628077    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:29.537641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:29.537608    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:23:30.644681 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:30.644638    2552 patch_prober.go:28] interesting pod/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 12:23:30.645070 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:30.644720    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 12:23:32.283101 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.283018    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"]
Sep 10 12:23:32.286578 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.286556    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.289966 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.289944    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-98aa83\"/\"kube-root-ca.crt\""
Sep 10 12:23:32.290595 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.290577    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-98aa83\"/\"default-dockercfg-7dk4x\""
Sep 10 12:23:32.291019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.290982    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-autores-98aa83\"/\"openshift-service-ca.crt\""
Sep 10 12:23:32.291153 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.291131    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-autores-98aa83\"/\"autores-model-98aa83-kserve-self-signed-certs\""
Sep 10 12:23:32.299491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.299468    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"]
Sep 10 12:23:32.419827 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.419785    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-dshm\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.420005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.419838    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-home\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.420005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.419859    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2hxb\" (UniqueName: \"kubernetes.io/projected/70ef661d-6e3c-4bc6-9983-515e76ab13aa-kube-api-access-n2hxb\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.420005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.419945    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tls-certs\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.420005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.419974    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-model-cache\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.420138 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.420058    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tmp-dir\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521169 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521134    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tls-certs\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521169 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521170    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-model-cache\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521367 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521213    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tmp-dir\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521367 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521324    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-dshm\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521513 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521391    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-home\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521513 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521416    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2hxb\" (UniqueName: \"kubernetes.io/projected/70ef661d-6e3c-4bc6-9983-515e76ab13aa-kube-api-access-n2hxb\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521621    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tmp-dir\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-model-cache\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.521752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.521699    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-home\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.523592 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.523574    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-dshm\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.523847 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.523830    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tls-certs\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.529927 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.529905    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2hxb\" (UniqueName: \"kubernetes.io/projected/70ef661d-6e3c-4bc6-9983-515e76ab13aa-kube-api-access-n2hxb\") pod \"autores-model-98aa83-kserve-5db766d9dc-t6xcd\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") " pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.597217 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.597126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:32.691636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.691601    2552 generic.go:356] "Generic (PLEG): container finished" podID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerID="7f4e04176ae407270e6138d8857151438323e1590f7d8813316fc27585782b2c" exitCode=0
Sep 10 12:23:32.691788 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.691675    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" event={"ID":"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78","Type":"ContainerDied","Data":"7f4e04176ae407270e6138d8857151438323e1590f7d8813316fc27585782b2c"}
Sep 10 12:23:32.692108 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.692094    2552 scope.go:117] "RemoveContainer" containerID="7f4e04176ae407270e6138d8857151438323e1590f7d8813316fc27585782b2c"
Sep 10 12:23:32.693199 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.693169    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:23:32.693290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.693244    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:23:32.693290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.693278    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:23:32.872848 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:32.872818    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"]
Sep 10 12:23:32.874534 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:23:32.874504    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70ef661d_6e3c_4bc6_9983_515e76ab13aa.slice/crio-5eb46fe422fd64028ab2d770678efd8f69624971f66899058b7546e6065afcdb WatchSource:0}: Error finding container 5eb46fe422fd64028ab2d770678efd8f69624971f66899058b7546e6065afcdb: Status 404 returned error can't find the container with id 5eb46fe422fd64028ab2d770678efd8f69624971f66899058b7546e6065afcdb
Sep 10 12:23:33.701022 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:33.700975    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" event={"ID":"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78","Type":"ContainerStarted","Data":"acab6ceb68eb38ad1c12058fde28747054f1eea0e637143c9a7ac92093eb9140"}
Sep 10 12:23:33.703971 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:33.703936    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd" event={"ID":"70ef661d-6e3c-4bc6-9983-515e76ab13aa","Type":"ContainerStarted","Data":"6be0127d1daf2cb100fc6f5942afe694e027f50f7a2e3b3e1350155cd4ddbbad"}
Sep 10 12:23:33.704093 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:33.703986    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd" event={"ID":"70ef661d-6e3c-4bc6-9983-515e76ab13aa","Type":"ContainerStarted","Data":"5eb46fe422fd64028ab2d770678efd8f69624971f66899058b7546e6065afcdb"}
Sep 10 12:23:33.755030 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:33.754976    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd" podStartSLOduration=1.754960626 podStartE2EDuration="1.754960626s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:23:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:23:33.754028979 +0000 UTC m=+824.950544680" watchObservedRunningTime="2026-09-10 12:23:33.754960626 +0000 UTC m=+824.951476326"
Sep 10 12:23:34.077696 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:34.077608    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:34.077849 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:34.077815    2552 patch_prober.go:28] interesting pod/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 12:23:34.077894 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:34.077874    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 12:23:35.077230 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:35.077199    2552 patch_prober.go:28] interesting pod/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 12:23:35.077632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:35.077260    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 12:23:36.077544 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:36.077508    2552 patch_prober.go:28] interesting pod/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 12:23:36.077969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:36.077566    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 12:23:37.077831 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:37.077789    2552 patch_prober.go:28] interesting pod/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 12:23:37.078184 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:37.077881    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 12:23:38.076821 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:38.076787    2552 patch_prober.go:28] interesting pod/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Sep 10 12:23:38.077018 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:38.076842    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Sep 10 12:23:39.080381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:39.080355    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:39.080789 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:39.080768    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:39.081404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:39.081384    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:23:42.597740 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:42.597695    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:42.597740 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:42.597752    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:42.610636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:42.610613    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:42.775511 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:42.775480    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:23:46.352232 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:46.352195    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:23:52.358103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.358057    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"]
Sep 10 12:23:52.363220 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.363200    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.371471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.371418    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7576a1\"/\"kube-root-ca.crt\""
Sep 10 12:23:52.372337 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.372312    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-7576a1\"/\"openshift-service-ca.crt\""
Sep 10 12:23:52.372485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.372371    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7576a1\"/\"default-dockercfg-9pl4t\""
Sep 10 12:23:52.372485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.372314    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-7576a1\"/\"test-model-7576a1-kserve-self-signed-certs\""
Sep 10 12:23:52.384263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.384237    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"]
Sep 10 12:23:52.504045 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.504010    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-home\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.504218 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.504066    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-model-cache\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.504218 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.504112    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-dshm\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.504218 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.504156    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2588855a-b69f-44fb-9ab7-45fafa86a83e-tls-certs\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.504218 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.504191    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-tmp-dir\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.504385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.504284    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-45hhp\" (UniqueName: \"kubernetes.io/projected/2588855a-b69f-44fb-9ab7-45fafa86a83e-kube-api-access-45hhp\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605363 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605331    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2588855a-b69f-44fb-9ab7-45fafa86a83e-tls-certs\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605379    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-tmp-dir\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605481    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-45hhp\" (UniqueName: \"kubernetes.io/projected/2588855a-b69f-44fb-9ab7-45fafa86a83e-kube-api-access-45hhp\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605541    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-home\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605587    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-model-cache\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605619    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-dshm\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605838 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605792    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-tmp-dir\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.605955 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.605875    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-home\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.606038 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.606018    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-model-cache\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.608052 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.608034    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2588855a-b69f-44fb-9ab7-45fafa86a83e-tls-certs\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.608242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.608196    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-dshm\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.620213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.620190    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-45hhp\" (UniqueName: \"kubernetes.io/projected/2588855a-b69f-44fb-9ab7-45fafa86a83e-kube-api-access-45hhp\") pod \"test-model-7576a1-kserve-9f9ffd955-vxhf4\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") " pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.673067 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.673036    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:23:52.949458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:52.949412    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"]
Sep 10 12:23:52.952262 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:23:52.952238    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2588855a_b69f_44fb_9ab7_45fafa86a83e.slice/crio-b5e091f8cd06f623a744c278d8acf9099c6bcb452eca123ec4c843fa5b34e59b WatchSource:0}: Error finding container b5e091f8cd06f623a744c278d8acf9099c6bcb452eca123ec4c843fa5b34e59b: Status 404 returned error can't find the container with id b5e091f8cd06f623a744c278d8acf9099c6bcb452eca123ec4c843fa5b34e59b
Sep 10 12:23:53.825720 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:53.825680    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4" event={"ID":"2588855a-b69f-44fb-9ab7-45fafa86a83e","Type":"ContainerStarted","Data":"359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e"}
Sep 10 12:23:53.826078 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:53.825726    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4" event={"ID":"2588855a-b69f-44fb-9ab7-45fafa86a83e","Type":"ContainerStarted","Data":"b5e091f8cd06f623a744c278d8acf9099c6bcb452eca123ec4c843fa5b34e59b"}
Sep 10 12:23:54.155255 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:54.155206    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4" podStartSLOduration=2.155189842 podStartE2EDuration="2.155189842s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:23:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:23:53.858884795 +0000 UTC m=+845.055400497" watchObservedRunningTime="2026-09-10 12:23:54.155189842 +0000 UTC m=+845.351705541"
Sep 10 12:23:54.155653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:54.155637    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"]
Sep 10 12:23:55.247757 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.247729    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"]
Sep 10 12:23:55.248148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.247917    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg" podUID="760f0b88-0d50-42e0-932e-011ac99923a3" containerName="payload-processing" containerID="cri-o://26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0" gracePeriod=30
Sep 10 12:23:55.504910 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.504855    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:55.634180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.634149    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760f0b88-0d50-42e0-932e-011ac99923a3-plugins-config-volume\") pod \"760f0b88-0d50-42e0-932e-011ac99923a3\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") "
Sep 10 12:23:55.634355 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.634235    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fv47c\" (UniqueName: \"kubernetes.io/projected/760f0b88-0d50-42e0-932e-011ac99923a3-kube-api-access-fv47c\") pod \"760f0b88-0d50-42e0-932e-011ac99923a3\" (UID: \"760f0b88-0d50-42e0-932e-011ac99923a3\") "
Sep 10 12:23:55.634514 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.634491    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/760f0b88-0d50-42e0-932e-011ac99923a3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "760f0b88-0d50-42e0-932e-011ac99923a3" (UID: "760f0b88-0d50-42e0-932e-011ac99923a3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:23:55.636361 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.636332    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/760f0b88-0d50-42e0-932e-011ac99923a3-kube-api-access-fv47c" (OuterVolumeSpecName: "kube-api-access-fv47c") pod "760f0b88-0d50-42e0-932e-011ac99923a3" (UID: "760f0b88-0d50-42e0-932e-011ac99923a3"). InnerVolumeSpecName "kube-api-access-fv47c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:23:55.643095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.643072    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"]
Sep 10 12:23:55.643326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.643301    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy" containerID="cri-o://acab6ceb68eb38ad1c12058fde28747054f1eea0e637143c9a7ac92093eb9140" gracePeriod=30
Sep 10 12:23:55.734953 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.734921    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fv47c\" (UniqueName: \"kubernetes.io/projected/760f0b88-0d50-42e0-932e-011ac99923a3-kube-api-access-fv47c\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:23:55.734953 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.734947    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/760f0b88-0d50-42e0-932e-011ac99923a3-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:23:55.835400 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.835326    2552 generic.go:356] "Generic (PLEG): container finished" podID="760f0b88-0d50-42e0-932e-011ac99923a3" containerID="26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0" exitCode=0
Sep 10 12:23:55.835400 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.835373    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"
Sep 10 12:23:55.835569 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.835410    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg" event={"ID":"760f0b88-0d50-42e0-932e-011ac99923a3","Type":"ContainerDied","Data":"26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0"}
Sep 10 12:23:55.835569 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.835474    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg" event={"ID":"760f0b88-0d50-42e0-932e-011ac99923a3","Type":"ContainerDied","Data":"868b6f3c621248112d4f542b7e89b1fba06475f10edaa8dc33ff91b9d25be67f"}
Sep 10 12:23:55.835569 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.835496    2552 scope.go:117] "RemoveContainer" containerID="26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0"
Sep 10 12:23:55.844703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.844688    2552 scope.go:117] "RemoveContainer" containerID="26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0"
Sep 10 12:23:55.844948 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:23:55.844929    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0\": container with ID starting with 26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0 not found: ID does not exist" containerID="26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0"
Sep 10 12:23:55.845010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.844957    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0"} err="failed to get container status \"26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0\": rpc error: code = NotFound desc = could not find container \"26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0\": container with ID starting with 26eec5d6c50dbf31026133cc982bc9c17e7f150cc400bfad548b8d91c7c6d9c0 not found: ID does not exist"
Sep 10 12:23:55.859501 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.859483    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"]
Sep 10 12:23:55.871033 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:55.871011    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-41f030dc-5b78f884c4-982bg"]
Sep 10 12:23:57.243298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:23:57.243258    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="760f0b88-0d50-42e0-932e-011ac99923a3" path="/var/lib/kubelet/pods/760f0b88-0d50-42e0-932e-011ac99923a3/volumes"
Sep 10 12:24:00.758408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:00.758379    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"]
Sep 10 12:24:00.863795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:00.863765    2552 generic.go:356] "Generic (PLEG): container finished" podID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerID="acab6ceb68eb38ad1c12058fde28747054f1eea0e637143c9a7ac92093eb9140" exitCode=0
Sep 10 12:24:00.863970 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:00.863819    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" event={"ID":"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78","Type":"ContainerDied","Data":"acab6ceb68eb38ad1c12058fde28747054f1eea0e637143c9a7ac92093eb9140"}
Sep 10 12:24:00.863970 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:00.863870    2552 scope.go:117] "RemoveContainer" containerID="7f4e04176ae407270e6138d8857151438323e1590f7d8813316fc27585782b2c"
Sep 10 12:24:00.903599 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:00.903576    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:24:01.085263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085172    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istiod-ca-cert\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085244    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-certs\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085274    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-token\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085303    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-envoy\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085352    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-data\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085378    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-socket\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085420    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-podinfo\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085527    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-credential-socket\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085539    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:01.085769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085560    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4ssqk\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-kube-api-access-4ssqk\") pod \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\" (UID: \"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78\") "
Sep 10 12:24:01.085769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085597    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:24:01.085769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085684    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-data" (OuterVolumeSpecName: "istio-data") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:01.086039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085824    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:01.086039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085917    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:01.086039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.085974    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.086039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.086024    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.086292 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.086037    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.087773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.087750    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:01.087880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.087770    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:24:01.087880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.087798    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-kube-api-access-4ssqk" (OuterVolumeSpecName: "kube-api-access-4ssqk") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "kube-api-access-4ssqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:01.087880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.087814    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-token" (OuterVolumeSpecName: "istio-token") pod "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" (UID: "e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:01.130070 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130039    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"]
Sep 10 12:24:01.130492 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130475    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy"
Sep 10 12:24:01.130593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130494    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy"
Sep 10 12:24:01.130593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130510    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="760f0b88-0d50-42e0-932e-011ac99923a3" containerName="payload-processing"
Sep 10 12:24:01.130593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130518    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="760f0b88-0d50-42e0-932e-011ac99923a3" containerName="payload-processing"
Sep 10 12:24:01.130752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130622    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy"
Sep 10 12:24:01.130752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130636    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy"
Sep 10 12:24:01.130752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.130646    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="760f0b88-0d50-42e0-932e-011ac99923a3" containerName="payload-processing"
Sep 10 12:24:01.133863 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.133842    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.138411 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.138384    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-67fd52b1\""
Sep 10 12:24:01.138547 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.138451    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-67fd52b1-dockercfg-gd8cn\""
Sep 10 12:24:01.148705 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.148678    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"]
Sep 10 12:24:01.187123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.187089    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.187123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.187114    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4ssqk\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-kube-api-access-4ssqk\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.187123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.187124    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.187356 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.187133    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.187356 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.187142    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.187356 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.187150    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:01.287674 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.287645    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zmmp4\" (UniqueName: \"kubernetes.io/projected/4be91536-3e7c-4846-bba7-7877e3026a80-kube-api-access-zmmp4\") pod \"payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") " pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.287674 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.287676    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91536-3e7c-4846-bba7-7877e3026a80-plugins-config-volume\") pod \"payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") " pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.389011 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.388985    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zmmp4\" (UniqueName: \"kubernetes.io/projected/4be91536-3e7c-4846-bba7-7877e3026a80-kube-api-access-zmmp4\") pod \"payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") " pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.389195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.389015    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91536-3e7c-4846-bba7-7877e3026a80-plugins-config-volume\") pod \"payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") " pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.389774 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.389751    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91536-3e7c-4846-bba7-7877e3026a80-plugins-config-volume\") pod \"payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") " pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.403572 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.403541    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zmmp4\" (UniqueName: \"kubernetes.io/projected/4be91536-3e7c-4846-bba7-7877e3026a80-kube-api-access-zmmp4\") pod \"payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") " pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.444053 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.444023    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.720308 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.720235    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"]
Sep 10 12:24:01.724233 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:24:01.724205    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4be91536_3e7c_4846_bba7_7877e3026a80.slice/crio-104201ea2396309a4c772f9e0d4a5f008c810fd1fb4c6c7263ff757f75042c99 WatchSource:0}: Error finding container 104201ea2396309a4c772f9e0d4a5f008c810fd1fb4c6c7263ff757f75042c99: Status 404 returned error can't find the container with id 104201ea2396309a4c772f9e0d4a5f008c810fd1fb4c6c7263ff757f75042c99
Sep 10 12:24:01.853199 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.853170    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"]
Sep 10 12:24:01.871063 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.871019    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk" event={"ID":"4be91536-3e7c-4846-bba7-7877e3026a80","Type":"ContainerStarted","Data":"e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f"}
Sep 10 12:24:01.871220 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.871084    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk" event={"ID":"4be91536-3e7c-4846-bba7-7877e3026a80","Type":"ContainerStarted","Data":"104201ea2396309a4c772f9e0d4a5f008c810fd1fb4c6c7263ff757f75042c99"}
Sep 10 12:24:01.871220 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.871114    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:01.873263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.873219    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"
Sep 10 12:24:01.873794 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.873217    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6" event={"ID":"e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78","Type":"ContainerDied","Data":"e1802fc2622848ff6f33d22deb665e21b86b43b2eb71d7cacb9849853f5945cf"}
Sep 10 12:24:01.873929 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.873830    2552 scope.go:117] "RemoveContainer" containerID="acab6ceb68eb38ad1c12058fde28747054f1eea0e637143c9a7ac92093eb9140"
Sep 10 12:24:01.892532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.892484    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk" podStartSLOduration=0.892469087 podStartE2EDuration="892.469087ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:24:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:24:01.891826323 +0000 UTC m=+853.088342037" watchObservedRunningTime="2026-09-10 12:24:01.892469087 +0000 UTC m=+853.088984811"
Sep 10 12:24:01.908621 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.908591    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"]
Sep 10 12:24:01.912883 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:01.912859    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-41f030dc-gw-openshift-default-789bc8784-b4mp6"]
Sep 10 12:24:02.673367 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:02.673325    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:24:02.673367 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:02.673373    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:24:02.686184 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:02.686155    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:24:02.890570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:02.890542    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:24:03.242962 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:03.242921    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" path="/var/lib/kubelet/pods/e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78/volumes"
Sep 10 12:24:04.272973 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:04.272941    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:24:09.175060 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.175007    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7" podUID="908d8713-d558-4ba5-a708-3f1a8c36c39f" containerName="main" containerID="cri-o://5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d" gracePeriod=45
Sep 10 12:24:09.415542 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.415522    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:24:09.461591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461527    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-tmp-dir\") pod \"908d8713-d558-4ba5-a708-3f1a8c36c39f\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") "
Sep 10 12:24:09.461591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461574    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-dshm\") pod \"908d8713-d558-4ba5-a708-3f1a8c36c39f\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") "
Sep 10 12:24:09.461765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461615    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4npz\" (UniqueName: \"kubernetes.io/projected/908d8713-d558-4ba5-a708-3f1a8c36c39f-kube-api-access-t4npz\") pod \"908d8713-d558-4ba5-a708-3f1a8c36c39f\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") "
Sep 10 12:24:09.461765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461633    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d8713-d558-4ba5-a708-3f1a8c36c39f-tls-certs\") pod \"908d8713-d558-4ba5-a708-3f1a8c36c39f\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") "
Sep 10 12:24:09.461765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461705    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-model-cache\") pod \"908d8713-d558-4ba5-a708-3f1a8c36c39f\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") "
Sep 10 12:24:09.461765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461738    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-home\") pod \"908d8713-d558-4ba5-a708-3f1a8c36c39f\" (UID: \"908d8713-d558-4ba5-a708-3f1a8c36c39f\") "
Sep 10 12:24:09.461765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461750    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "908d8713-d558-4ba5-a708-3f1a8c36c39f" (UID: "908d8713-d558-4ba5-a708-3f1a8c36c39f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:09.462025 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.461955    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-model-cache" (OuterVolumeSpecName: "model-cache") pod "908d8713-d558-4ba5-a708-3f1a8c36c39f" (UID: "908d8713-d558-4ba5-a708-3f1a8c36c39f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:09.462094 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.462055    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:09.462146 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.462073    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:09.462206 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.462153    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-home" (OuterVolumeSpecName: "home") pod "908d8713-d558-4ba5-a708-3f1a8c36c39f" (UID: "908d8713-d558-4ba5-a708-3f1a8c36c39f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:09.463928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.463896    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-dshm" (OuterVolumeSpecName: "dshm") pod "908d8713-d558-4ba5-a708-3f1a8c36c39f" (UID: "908d8713-d558-4ba5-a708-3f1a8c36c39f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:09.464018 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.463980    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/908d8713-d558-4ba5-a708-3f1a8c36c39f-kube-api-access-t4npz" (OuterVolumeSpecName: "kube-api-access-t4npz") pod "908d8713-d558-4ba5-a708-3f1a8c36c39f" (UID: "908d8713-d558-4ba5-a708-3f1a8c36c39f"). InnerVolumeSpecName "kube-api-access-t4npz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:09.464055 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.464007    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/908d8713-d558-4ba5-a708-3f1a8c36c39f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "908d8713-d558-4ba5-a708-3f1a8c36c39f" (UID: "908d8713-d558-4ba5-a708-3f1a8c36c39f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:24:09.562750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.562721    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:09.562750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.562745    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/908d8713-d558-4ba5-a708-3f1a8c36c39f-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:09.562886 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.562758    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4npz\" (UniqueName: \"kubernetes.io/projected/908d8713-d558-4ba5-a708-3f1a8c36c39f-kube-api-access-t4npz\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:09.562886 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.562773    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/908d8713-d558-4ba5-a708-3f1a8c36c39f-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:09.916885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.916853    2552 generic.go:356] "Generic (PLEG): container finished" podID="908d8713-d558-4ba5-a708-3f1a8c36c39f" containerID="5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d" exitCode=0
Sep 10 12:24:09.917029 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.916896    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"
Sep 10 12:24:09.917029 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.916905    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7" event={"ID":"908d8713-d558-4ba5-a708-3f1a8c36c39f","Type":"ContainerDied","Data":"5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d"}
Sep 10 12:24:09.917029 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.916950    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7" event={"ID":"908d8713-d558-4ba5-a708-3f1a8c36c39f","Type":"ContainerDied","Data":"0375c08cf07eaccaaa99c4cba0b88570c33d49c34e88ff9ad3986b14909e9013"}
Sep 10 12:24:09.917029 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.916967    2552 scope.go:117] "RemoveContainer" containerID="5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d"
Sep 10 12:24:09.928207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.928185    2552 scope.go:117] "RemoveContainer" containerID="5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d"
Sep 10 12:24:09.928701 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:24:09.928677    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d\": container with ID starting with 5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d not found: ID does not exist" containerID="5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d"
Sep 10 12:24:09.928798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.928713    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d"} err="failed to get container status \"5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d\": rpc error: code = NotFound desc = could not find container \"5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d\": container with ID starting with 5f9617ab431a452b1c01bb4969776c0796a15476886794158aacebde9effcb8d not found: ID does not exist"
Sep 10 12:24:09.940721 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.940701    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"]
Sep 10 12:24:09.942261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:09.942239    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-be65aff6-kserve-6d59d8499d-6xrp7"]
Sep 10 12:24:11.243151 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:11.243108    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="908d8713-d558-4ba5-a708-3f1a8c36c39f" path="/var/lib/kubelet/pods/908d8713-d558-4ba5-a708-3f1a8c36c39f/volumes"
Sep 10 12:24:12.034452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.034403    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"]
Sep 10 12:24:12.034677 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.034654    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n" podUID="70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" containerName="payload-processing" containerID="cri-o://52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9" gracePeriod=30
Sep 10 12:24:12.048988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.048964    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"]
Sep 10 12:24:12.049194 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.049142    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb" podUID="d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" containerName="payload-pre-processing" containerID="cri-o://171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd" gracePeriod=30
Sep 10 12:24:12.320031 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.320005    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:24:12.322760 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.322738    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:24:12.388415 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.388381    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-plugins-config-volume\") pod \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") "
Sep 10 12:24:12.388585 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.388451    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4vsx2\" (UniqueName: \"kubernetes.io/projected/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-kube-api-access-4vsx2\") pod \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") "
Sep 10 12:24:12.388585 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.388483    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvjrz\" (UniqueName: \"kubernetes.io/projected/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-kube-api-access-wvjrz\") pod \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\" (UID: \"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6\") "
Sep 10 12:24:12.388709 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.388603    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-plugins-config-volume\") pod \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\" (UID: \"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4\") "
Sep 10 12:24:12.388979 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.388947    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" (UID: "70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:24:12.388979 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.388967    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" (UID: "d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:24:12.390851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.390820    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-kube-api-access-wvjrz" (OuterVolumeSpecName: "kube-api-access-wvjrz") pod "70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" (UID: "70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6"). InnerVolumeSpecName "kube-api-access-wvjrz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:12.391007 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.390985    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-kube-api-access-4vsx2" (OuterVolumeSpecName: "kube-api-access-4vsx2") pod "d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" (UID: "d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4"). InnerVolumeSpecName "kube-api-access-4vsx2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:12.490112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.490072    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:12.490112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.490105    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:12.490112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.490119    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4vsx2\" (UniqueName: \"kubernetes.io/projected/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4-kube-api-access-4vsx2\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:12.490345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.490133    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvjrz\" (UniqueName: \"kubernetes.io/projected/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6-kube-api-access-wvjrz\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:12.881875 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.881844    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:12.934108 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.934075    2552 generic.go:356] "Generic (PLEG): container finished" podID="70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" containerID="52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9" exitCode=0
Sep 10 12:24:12.934267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.934128    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"
Sep 10 12:24:12.934267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.934163    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n" event={"ID":"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6","Type":"ContainerDied","Data":"52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9"}
Sep 10 12:24:12.934267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.934223    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n" event={"ID":"70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6","Type":"ContainerDied","Data":"2b3c24658de5e7f651ede04c31008923d9352399cd57e5bc1da3ac1d8783a438"}
Sep 10 12:24:12.934267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.934253    2552 scope.go:117] "RemoveContainer" containerID="52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9"
Sep 10 12:24:12.935773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.935749    2552 generic.go:356] "Generic (PLEG): container finished" podID="d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" containerID="171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd" exitCode=0
Sep 10 12:24:12.935873 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.935793    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"
Sep 10 12:24:12.935873 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.935801    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb" event={"ID":"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4","Type":"ContainerDied","Data":"171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd"}
Sep 10 12:24:12.935873 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.935845    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb" event={"ID":"d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4","Type":"ContainerDied","Data":"f5f9279f704f10fc9d4fb2ddb37d0004bb14209701adc1974dc3212f994e202a"}
Sep 10 12:24:12.945014 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.944995    2552 scope.go:117] "RemoveContainer" containerID="52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9"
Sep 10 12:24:12.945371 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:24:12.945346    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9\": container with ID starting with 52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9 not found: ID does not exist" containerID="52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9"
Sep 10 12:24:12.945469 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.945383    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9"} err="failed to get container status \"52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9\": rpc error: code = NotFound desc = could not find container \"52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9\": container with ID starting with 52557436104db3a733ae84e8b1cf3b47d5423c49fa5cc4fbcfacb8152d1328e9 not found: ID does not exist"
Sep 10 12:24:12.945469 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.945408    2552 scope.go:117] "RemoveContainer" containerID="171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd"
Sep 10 12:24:12.953497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.953479    2552 scope.go:117] "RemoveContainer" containerID="171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd"
Sep 10 12:24:12.953762 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:24:12.953740    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd\": container with ID starting with 171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd not found: ID does not exist" containerID="171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd"
Sep 10 12:24:12.953864 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.953771    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd"} err="failed to get container status \"171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd\": rpc error: code = NotFound desc = could not find container \"171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd\": container with ID starting with 171ce850c95614d699466ed563f705e302aa826b5a2eba60860c095387b2d5fd not found: ID does not exist"
Sep 10 12:24:12.963157 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.963134    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"]
Sep 10 12:24:12.973541 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.973403    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-98aa83-7984bcbc7c-qlq9n"]
Sep 10 12:24:12.990458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.990419    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"]
Sep 10 12:24:12.991949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:12.991928    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-98aa83-55467cbff4-dxhjb"]
Sep 10 12:24:13.242565 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:13.242493    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" path="/var/lib/kubelet/pods/70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6/volumes"
Sep 10 12:24:13.242876 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:13.242863    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" path="/var/lib/kubelet/pods/d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4/volumes"
Sep 10 12:24:15.778601 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:15.778555    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd" podUID="70ef661d-6e3c-4bc6-9983-515e76ab13aa" containerName="main" containerID="cri-o://6be0127d1daf2cb100fc6f5942afe694e027f50f7a2e3b3e1350155cd4ddbbad" gracePeriod=45
Sep 10 12:24:15.954699 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:15.954669    2552 generic.go:356] "Generic (PLEG): container finished" podID="70ef661d-6e3c-4bc6-9983-515e76ab13aa" containerID="6be0127d1daf2cb100fc6f5942afe694e027f50f7a2e3b3e1350155cd4ddbbad" exitCode=0
Sep 10 12:24:15.954847 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:15.954743    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd" event={"ID":"70ef661d-6e3c-4bc6-9983-515e76ab13aa","Type":"ContainerDied","Data":"6be0127d1daf2cb100fc6f5942afe694e027f50f7a2e3b3e1350155cd4ddbbad"}
Sep 10 12:24:16.028751 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.028688    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:24:16.121105 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121078    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-dshm\") pod \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") "
Sep 10 12:24:16.121258 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121117    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tmp-dir\") pod \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") "
Sep 10 12:24:16.121258 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121156    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-model-cache\") pod \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") "
Sep 10 12:24:16.121258 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121182    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-home\") pod \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") "
Sep 10 12:24:16.121393 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121292    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n2hxb\" (UniqueName: \"kubernetes.io/projected/70ef661d-6e3c-4bc6-9983-515e76ab13aa-kube-api-access-n2hxb\") pod \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") "
Sep 10 12:24:16.121393 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121336    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tls-certs\") pod \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\" (UID: \"70ef661d-6e3c-4bc6-9983-515e76ab13aa\") "
Sep 10 12:24:16.121523 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121392    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-home" (OuterVolumeSpecName: "home") pod "70ef661d-6e3c-4bc6-9983-515e76ab13aa" (UID: "70ef661d-6e3c-4bc6-9983-515e76ab13aa"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:16.121523 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121450    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70ef661d-6e3c-4bc6-9983-515e76ab13aa" (UID: "70ef661d-6e3c-4bc6-9983-515e76ab13aa"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:16.121523 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121483    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-model-cache" (OuterVolumeSpecName: "model-cache") pod "70ef661d-6e3c-4bc6-9983-515e76ab13aa" (UID: "70ef661d-6e3c-4bc6-9983-515e76ab13aa"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:16.121729 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121708    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:16.121729 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121728    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:16.122070 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.121738    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:16.123561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.123539    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70ef661d-6e3c-4bc6-9983-515e76ab13aa-kube-api-access-n2hxb" (OuterVolumeSpecName: "kube-api-access-n2hxb") pod "70ef661d-6e3c-4bc6-9983-515e76ab13aa" (UID: "70ef661d-6e3c-4bc6-9983-515e76ab13aa"). InnerVolumeSpecName "kube-api-access-n2hxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:16.123561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.123540    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70ef661d-6e3c-4bc6-9983-515e76ab13aa" (UID: "70ef661d-6e3c-4bc6-9983-515e76ab13aa"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:24:16.123732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.123547    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-dshm" (OuterVolumeSpecName: "dshm") pod "70ef661d-6e3c-4bc6-9983-515e76ab13aa" (UID: "70ef661d-6e3c-4bc6-9983-515e76ab13aa"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:16.223147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.223111    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70ef661d-6e3c-4bc6-9983-515e76ab13aa-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:16.223147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.223141    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70ef661d-6e3c-4bc6-9983-515e76ab13aa-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:16.223147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.223151    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2hxb\" (UniqueName: \"kubernetes.io/projected/70ef661d-6e3c-4bc6-9983-515e76ab13aa-kube-api-access-n2hxb\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:16.881386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.881328    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f" podUID="46936de5-52a4-4733-9634-879f7fd35dfc" containerName="main" containerID="cri-o://023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36" gracePeriod=45
Sep 10 12:24:16.960447 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.960403    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"
Sep 10 12:24:16.960584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.960453    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd" event={"ID":"70ef661d-6e3c-4bc6-9983-515e76ab13aa","Type":"ContainerDied","Data":"5eb46fe422fd64028ab2d770678efd8f69624971f66899058b7546e6065afcdb"}
Sep 10 12:24:16.960584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.960503    2552 scope.go:117] "RemoveContainer" containerID="6be0127d1daf2cb100fc6f5942afe694e027f50f7a2e3b3e1350155cd4ddbbad"
Sep 10 12:24:16.994280 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:16.994254    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"]
Sep 10 12:24:17.008129 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.008102    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-autores-98aa83/autores-model-98aa83-kserve-5db766d9dc-t6xcd"]
Sep 10 12:24:17.129040 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.129008    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:24:17.230259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230162    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b2xtd\" (UniqueName: \"kubernetes.io/projected/46936de5-52a4-4733-9634-879f7fd35dfc-kube-api-access-b2xtd\") pod \"46936de5-52a4-4733-9634-879f7fd35dfc\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") "
Sep 10 12:24:17.230259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230223    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-dshm\") pod \"46936de5-52a4-4733-9634-879f7fd35dfc\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") "
Sep 10 12:24:17.230524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230332    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-home\") pod \"46936de5-52a4-4733-9634-879f7fd35dfc\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") "
Sep 10 12:24:17.230524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230392    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46936de5-52a4-4733-9634-879f7fd35dfc-tls-certs\") pod \"46936de5-52a4-4733-9634-879f7fd35dfc\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") "
Sep 10 12:24:17.230524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230415    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-model-cache\") pod \"46936de5-52a4-4733-9634-879f7fd35dfc\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") "
Sep 10 12:24:17.230702 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230546    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-tmp-dir\") pod \"46936de5-52a4-4733-9634-879f7fd35dfc\" (UID: \"46936de5-52a4-4733-9634-879f7fd35dfc\") "
Sep 10 12:24:17.230702 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230670    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-model-cache" (OuterVolumeSpecName: "model-cache") pod "46936de5-52a4-4733-9634-879f7fd35dfc" (UID: "46936de5-52a4-4733-9634-879f7fd35dfc"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:17.230702 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230684    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-home" (OuterVolumeSpecName: "home") pod "46936de5-52a4-4733-9634-879f7fd35dfc" (UID: "46936de5-52a4-4733-9634-879f7fd35dfc"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:17.230828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230790    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "46936de5-52a4-4733-9634-879f7fd35dfc" (UID: "46936de5-52a4-4733-9634-879f7fd35dfc"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:17.230923 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230908    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:17.230969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230927    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:17.230969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.230938    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:17.232603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.232579    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/46936de5-52a4-4733-9634-879f7fd35dfc-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "46936de5-52a4-4733-9634-879f7fd35dfc" (UID: "46936de5-52a4-4733-9634-879f7fd35dfc"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:24:17.233011 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.232984    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-dshm" (OuterVolumeSpecName: "dshm") pod "46936de5-52a4-4733-9634-879f7fd35dfc" (UID: "46936de5-52a4-4733-9634-879f7fd35dfc"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:24:17.233011 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.232994    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/46936de5-52a4-4733-9634-879f7fd35dfc-kube-api-access-b2xtd" (OuterVolumeSpecName: "kube-api-access-b2xtd") pod "46936de5-52a4-4733-9634-879f7fd35dfc" (UID: "46936de5-52a4-4733-9634-879f7fd35dfc"). InnerVolumeSpecName "kube-api-access-b2xtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:17.241723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.241698    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70ef661d-6e3c-4bc6-9983-515e76ab13aa" path="/var/lib/kubelet/pods/70ef661d-6e3c-4bc6-9983-515e76ab13aa/volumes"
Sep 10 12:24:17.332066 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.332030    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/46936de5-52a4-4733-9634-879f7fd35dfc-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:17.332066 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.332060    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2xtd\" (UniqueName: \"kubernetes.io/projected/46936de5-52a4-4733-9634-879f7fd35dfc-kube-api-access-b2xtd\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:17.332066 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.332070    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/46936de5-52a4-4733-9634-879f7fd35dfc-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:17.966226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.966191    2552 generic.go:356] "Generic (PLEG): container finished" podID="46936de5-52a4-4733-9634-879f7fd35dfc" containerID="023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36" exitCode=0
Sep 10 12:24:17.966671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.966233    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"
Sep 10 12:24:17.966671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.966281    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f" event={"ID":"46936de5-52a4-4733-9634-879f7fd35dfc","Type":"ContainerDied","Data":"023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36"}
Sep 10 12:24:17.966671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.966325    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f" event={"ID":"46936de5-52a4-4733-9634-879f7fd35dfc","Type":"ContainerDied","Data":"411550a761503e3131f0731c85f8b5231aac132b66e9ffa7023f6fc63cd2b29e"}
Sep 10 12:24:17.966671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.966347    2552 scope.go:117] "RemoveContainer" containerID="023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36"
Sep 10 12:24:17.974636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.974619    2552 scope.go:117] "RemoveContainer" containerID="023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36"
Sep 10 12:24:17.974917 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:24:17.974897    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36\": container with ID starting with 023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36 not found: ID does not exist" containerID="023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36"
Sep 10 12:24:17.974993 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.974929    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36"} err="failed to get container status \"023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36\": rpc error: code = NotFound desc = could not find container \"023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36\": container with ID starting with 023e55d9308e8ea124aceb2f5d081eb3e394f5dfc035e8f95c9d70127d249e36 not found: ID does not exist"
Sep 10 12:24:17.995924 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:17.995894    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"]
Sep 10 12:24:18.004580 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:18.004557    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-be65aff6-kserve-d96b74456-nfj4f"]
Sep 10 12:24:19.242269 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:19.242238    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="46936de5-52a4-4733-9634-879f7fd35dfc" path="/var/lib/kubelet/pods/46936de5-52a4-4733-9634-879f7fd35dfc/volumes"
Sep 10 12:24:20.020604 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.020573    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:24:20.804500 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804474    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"]
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804799    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" containerName="payload-pre-processing"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804810    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" containerName="payload-pre-processing"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804821    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" containerName="payload-processing"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804828    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" containerName="payload-processing"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804836    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804841    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="e8a0d53e-f631-4c16-a5ff-f29c5b7a6e78" containerName="istio-proxy"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804851    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="908d8713-d558-4ba5-a708-3f1a8c36c39f" containerName="main"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804856    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="908d8713-d558-4ba5-a708-3f1a8c36c39f" containerName="main"
Sep 10 12:24:20.804862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804864    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="46936de5-52a4-4733-9634-879f7fd35dfc" containerName="main"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804869    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="46936de5-52a4-4733-9634-879f7fd35dfc" containerName="main"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804875    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70ef661d-6e3c-4bc6-9983-515e76ab13aa" containerName="main"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804880    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="70ef661d-6e3c-4bc6-9983-515e76ab13aa" containerName="main"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804944    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="46936de5-52a4-4733-9634-879f7fd35dfc" containerName="main"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804954    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="908d8713-d558-4ba5-a708-3f1a8c36c39f" containerName="main"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804960    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d71ef12f-063c-4bd8-b5b1-6dc0262ea2d4" containerName="payload-pre-processing"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804966    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="70a45bc5-aee1-4fd0-9cd4-cfa5adaf26d6" containerName="payload-processing"
Sep 10 12:24:20.805180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.804972    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="70ef661d-6e3c-4bc6-9983-515e76ab13aa" containerName="main"
Sep 10 12:24:20.808939 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.808921    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.816768 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.816739    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-f615cc-openshift-default-dockercfg-7rs7z\""
Sep 10 12:24:20.817368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.817349    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 12:24:20.838919 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.838893    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"]
Sep 10 12:24:20.860143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860116    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-envoy\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860143    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-token\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860165    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-certs\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860197    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istiod-ca-cert\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860213    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whpc2\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-kube-api-access-whpc2\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860236    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-credential-socket\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860332    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-socket\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860356    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-data\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.860485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.860382    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-podinfo\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.960847 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.960804    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-envoy\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.960847 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.960846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-token\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961023 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.960870    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-certs\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961023 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.960987    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istiod-ca-cert\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961146 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.961026    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whpc2\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-kube-api-access-whpc2\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961146 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.961069    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-credential-socket\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961146 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.961143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-socket\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961306 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.961168    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-data\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961306 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.961185    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-certs\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.961306 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.961200    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-podinfo\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.962396 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.962170    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-data\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.962396 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.962367    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istiod-ca-cert\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.962588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.962372    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-credential-socket\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.962588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.962398    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-socket\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.968160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.964259    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-envoy\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.968160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.964771    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-podinfo\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.972963 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.972937    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-whpc2\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-kube-api-access-whpc2\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:20.975226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:20.975203    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-token\") pod \"e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") " pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:21.119198 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:21.119175    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:21.416944 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:21.416784    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"]
Sep 10 12:24:21.419011 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:24:21.418987    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f209ffd_75c4_4c14_b404_0e5df8f3e7c8.slice/crio-eed1839be5bde7e0b59a6b64a73cdcc382e2be6dae29406676071c256b1dfe8d WatchSource:0}: Error finding container eed1839be5bde7e0b59a6b64a73cdcc382e2be6dae29406676071c256b1dfe8d: Status 404 returned error can't find the container with id eed1839be5bde7e0b59a6b64a73cdcc382e2be6dae29406676071c256b1dfe8d
Sep 10 12:24:21.421392 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:21.421356    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:24:21.421479 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:21.421457    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:24:21.421518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:21.421503    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:24:21.990741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:21.990708    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" event={"ID":"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8","Type":"ContainerStarted","Data":"064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd"}
Sep 10 12:24:21.991102 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:21.990758    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" event={"ID":"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8","Type":"ContainerStarted","Data":"eed1839be5bde7e0b59a6b64a73cdcc382e2be6dae29406676071c256b1dfe8d"}
Sep 10 12:24:22.042194 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:22.042146    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podStartSLOduration=2.042129727 podStartE2EDuration="2.042129727s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:24:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:24:22.039762868 +0000 UTC m=+873.236278568" watchObservedRunningTime="2026-09-10 12:24:22.042129727 +0000 UTC m=+873.238645426"
Sep 10 12:24:22.120132 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:22.120098    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:22.121704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:22.121681    2552 patch_prober.go:28] interesting pod/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:24:22.121775 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:22.121739    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:24:23.123785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:23.123761    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:24.002272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.002239    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:24.003307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.003288    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:24.751332 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.751302    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"]
Sep 10 12:24:24.754791 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.754775    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"
Sep 10 12:24:24.757456 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.757415    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:24:24.757751 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.757735    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:24:24.766660 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.766639    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"]
Sep 10 12:24:24.796906 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.796881    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1779503-4d10-4f11-8d1f-88fa1ade5f8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8\" (UID: \"a1779503-4d10-4f11-8d1f-88fa1ade5f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"
Sep 10 12:24:24.897516 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.897485    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1779503-4d10-4f11-8d1f-88fa1ade5f8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8\" (UID: \"a1779503-4d10-4f11-8d1f-88fa1ade5f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"
Sep 10 12:24:24.898235 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:24.898211    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1779503-4d10-4f11-8d1f-88fa1ade5f8d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8\" (UID: \"a1779503-4d10-4f11-8d1f-88fa1ade5f8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"
Sep 10 12:24:25.065700 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:25.065643    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"
Sep 10 12:24:25.344667 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:25.344638    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"]
Sep 10 12:24:25.347061 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:24:25.347034    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1779503_4d10_4f11_8d1f_88fa1ade5f8d.slice/crio-a3c555deae93ed74e71418a894283c93debbee9838e59c49b0b8e35fcaf6c126 WatchSource:0}: Error finding container a3c555deae93ed74e71418a894283c93debbee9838e59c49b0b8e35fcaf6c126: Status 404 returned error can't find the container with id a3c555deae93ed74e71418a894283c93debbee9838e59c49b0b8e35fcaf6c126
Sep 10 12:24:26.012368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.012337    2552 generic.go:356] "Generic (PLEG): container finished" podID="a1779503-4d10-4f11-8d1f-88fa1ade5f8d" containerID="0692518e07c18434fd82cbfd6ca2581446047cddcfd08fca33c85a0c9d824093" exitCode=0
Sep 10 12:24:26.012745 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.012403    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8" event={"ID":"a1779503-4d10-4f11-8d1f-88fa1ade5f8d","Type":"ContainerDied","Data":"0692518e07c18434fd82cbfd6ca2581446047cddcfd08fca33c85a0c9d824093"}
Sep 10 12:24:26.012745 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.012460    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8" event={"ID":"a1779503-4d10-4f11-8d1f-88fa1ade5f8d","Type":"ContainerStarted","Data":"a3c555deae93ed74e71418a894283c93debbee9838e59c49b0b8e35fcaf6c126"}
Sep 10 12:24:26.511228 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.511193    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"]
Sep 10 12:24:26.514626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.514610    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.517406 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.517387    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-f615cc\""
Sep 10 12:24:26.525319 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.525299    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"]
Sep 10 12:24:26.528611 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.528594    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:26.531064 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.531048    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-f615cc-dockercfg-5sr2l\""
Sep 10 12:24:26.533697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.533673    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"]
Sep 10 12:24:26.550483 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.550452    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"]
Sep 10 12:24:26.615094 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.615055    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") " pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:26.615094 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.615090    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m4j8v\" (UniqueName: \"kubernetes.io/projected/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-kube-api-access-m4j8v\") pod \"payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") " pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:26.615344 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.615154    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b9xdh\" (UniqueName: \"kubernetes.io/projected/3e490442-3318-4958-91de-e30c13ed4f01-kube-api-access-b9xdh\") pod \"payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.615344 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.615276    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e490442-3318-4958-91de-e30c13ed4f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.715845 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.715807    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e490442-3318-4958-91de-e30c13ed4f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.716052 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.715866    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") " pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:26.716052 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.715893    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m4j8v\" (UniqueName: \"kubernetes.io/projected/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-kube-api-access-m4j8v\") pod \"payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") " pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:26.716052 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.715952    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b9xdh\" (UniqueName: \"kubernetes.io/projected/3e490442-3318-4958-91de-e30c13ed4f01-kube-api-access-b9xdh\") pod \"payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.716668 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.716646    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e490442-3318-4958-91de-e30c13ed4f01-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.716716 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.716692    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-plugins-config-volume\") pod \"payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") " pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:26.724994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.724963    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b9xdh\" (UniqueName: \"kubernetes.io/projected/3e490442-3318-4958-91de-e30c13ed4f01-kube-api-access-b9xdh\") pod \"payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.725510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.725490    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m4j8v\" (UniqueName: \"kubernetes.io/projected/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-kube-api-access-m4j8v\") pod \"payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") " pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:26.825534 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.825461    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:26.838708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:26.838686    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:27.106373 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:27.106300    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"]
Sep 10 12:24:27.110489 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:24:27.110461    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3e490442_3318_4958_91de_e30c13ed4f01.slice/crio-61b3efbd7bc80419c4949845cb38f55c433b4390a3096f95e9c1b41f4ffaf0b4 WatchSource:0}: Error finding container 61b3efbd7bc80419c4949845cb38f55c433b4390a3096f95e9c1b41f4ffaf0b4: Status 404 returned error can't find the container with id 61b3efbd7bc80419c4949845cb38f55c433b4390a3096f95e9c1b41f4ffaf0b4
Sep 10 12:24:27.129397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:27.129360    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"]
Sep 10 12:24:27.131235 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:24:27.131214    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d8755b4_2d5e_4fdc_b0d6_1663c3ff959c.slice/crio-4fafa27d1ce2898af1199dad80ee5cf3b8372062546ee478cb390a3a0037f8b7 WatchSource:0}: Error finding container 4fafa27d1ce2898af1199dad80ee5cf3b8372062546ee478cb390a3a0037f8b7: Status 404 returned error can't find the container with id 4fafa27d1ce2898af1199dad80ee5cf3b8372062546ee478cb390a3a0037f8b7
Sep 10 12:24:27.133346 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:27.133328    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"
Sep 10 12:24:27.220869 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:27.220835    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1779503-4d10-4f11-8d1f-88fa1ade5f8d-maas-api-service-ca\") pod \"a1779503-4d10-4f11-8d1f-88fa1ade5f8d\" (UID: \"a1779503-4d10-4f11-8d1f-88fa1ade5f8d\") "
Sep 10 12:24:27.221331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:27.221262    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1779503-4d10-4f11-8d1f-88fa1ade5f8d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a1779503-4d10-4f11-8d1f-88fa1ade5f8d" (UID: "a1779503-4d10-4f11-8d1f-88fa1ade5f8d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:24:27.322557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:27.322520    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a1779503-4d10-4f11-8d1f-88fa1ade5f8d-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:28.025310 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.025225    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"
Sep 10 12:24:28.025479 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.025219    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8" event={"ID":"a1779503-4d10-4f11-8d1f-88fa1ade5f8d","Type":"ContainerDied","Data":"a3c555deae93ed74e71418a894283c93debbee9838e59c49b0b8e35fcaf6c126"}
Sep 10 12:24:28.025479 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.025357    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a3c555deae93ed74e71418a894283c93debbee9838e59c49b0b8e35fcaf6c126"
Sep 10 12:24:28.027234 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.027205    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr" event={"ID":"3e490442-3318-4958-91de-e30c13ed4f01","Type":"ContainerStarted","Data":"b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b"}
Sep 10 12:24:28.027379 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.027260    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:28.027379 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.027285    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr" event={"ID":"3e490442-3318-4958-91de-e30c13ed4f01","Type":"ContainerStarted","Data":"61b3efbd7bc80419c4949845cb38f55c433b4390a3096f95e9c1b41f4ffaf0b4"}
Sep 10 12:24:28.029119 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.029101    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s" event={"ID":"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c","Type":"ContainerStarted","Data":"c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e"}
Sep 10 12:24:28.029246 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.029131    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s" event={"ID":"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c","Type":"ContainerStarted","Data":"4fafa27d1ce2898af1199dad80ee5cf3b8372062546ee478cb390a3a0037f8b7"}
Sep 10 12:24:28.029246 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.029233    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:28.045953 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.045900    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr" podStartSLOduration=2.045881623 podStartE2EDuration="2.045881623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:24:28.044777493 +0000 UTC m=+879.241293193" watchObservedRunningTime="2026-09-10 12:24:28.045881623 +0000 UTC m=+879.242397324"
Sep 10 12:24:28.072489 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:28.072413    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s" podStartSLOduration=2.072396146 podStartE2EDuration="2.072396146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:24:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:24:28.070921904 +0000 UTC m=+879.267437607" watchObservedRunningTime="2026-09-10 12:24:28.072396146 +0000 UTC m=+879.268911849"
Sep 10 12:24:31.266216 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.266183    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"]
Sep 10 12:24:31.266679 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.266413    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk" podUID="4be91536-3e7c-4846-bba7-7877e3026a80" containerName="payload-processing" containerID="cri-o://e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f" gracePeriod=30
Sep 10 12:24:31.531318 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.531291    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:31.663350 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.663311    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91536-3e7c-4846-bba7-7877e3026a80-plugins-config-volume\") pod \"4be91536-3e7c-4846-bba7-7877e3026a80\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") "
Sep 10 12:24:31.663593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.663401    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-zmmp4\" (UniqueName: \"kubernetes.io/projected/4be91536-3e7c-4846-bba7-7877e3026a80-kube-api-access-zmmp4\") pod \"4be91536-3e7c-4846-bba7-7877e3026a80\" (UID: \"4be91536-3e7c-4846-bba7-7877e3026a80\") "
Sep 10 12:24:31.663706 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.663682    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4be91536-3e7c-4846-bba7-7877e3026a80-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4be91536-3e7c-4846-bba7-7877e3026a80" (UID: "4be91536-3e7c-4846-bba7-7877e3026a80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:24:31.663798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.663783    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4be91536-3e7c-4846-bba7-7877e3026a80-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:31.665575 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.665555    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4be91536-3e7c-4846-bba7-7877e3026a80-kube-api-access-zmmp4" (OuterVolumeSpecName: "kube-api-access-zmmp4") pod "4be91536-3e7c-4846-bba7-7877e3026a80" (UID: "4be91536-3e7c-4846-bba7-7877e3026a80"). InnerVolumeSpecName "kube-api-access-zmmp4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:24:31.688483 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.688452    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"]
Sep 10 12:24:31.698709 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.698684    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-67fd52b1-gh6w8"]
Sep 10 12:24:31.764508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:31.764472    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zmmp4\" (UniqueName: \"kubernetes.io/projected/4be91536-3e7c-4846-bba7-7877e3026a80-kube-api-access-zmmp4\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:24:32.049488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.049452    2552 generic.go:356] "Generic (PLEG): container finished" podID="4be91536-3e7c-4846-bba7-7877e3026a80" containerID="e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f" exitCode=0
Sep 10 12:24:32.049689 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.049507    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"
Sep 10 12:24:32.049689 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.049541    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk" event={"ID":"4be91536-3e7c-4846-bba7-7877e3026a80","Type":"ContainerDied","Data":"e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f"}
Sep 10 12:24:32.049689 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.049595    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk" event={"ID":"4be91536-3e7c-4846-bba7-7877e3026a80","Type":"ContainerDied","Data":"104201ea2396309a4c772f9e0d4a5f008c810fd1fb4c6c7263ff757f75042c99"}
Sep 10 12:24:32.049689 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.049621    2552 scope.go:117] "RemoveContainer" containerID="e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f"
Sep 10 12:24:32.058822 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.058805    2552 scope.go:117] "RemoveContainer" containerID="e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f"
Sep 10 12:24:32.059083 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:24:32.059066    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f\": container with ID starting with e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f not found: ID does not exist" containerID="e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f"
Sep 10 12:24:32.059145 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.059095    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f"} err="failed to get container status \"e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f\": rpc error: code = NotFound desc = could not find container \"e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f\": container with ID starting with e3bc40b5150a93e601f1c635cb0cc29616da269c2b2a4d8f4c727a7593ef8b1f not found: ID does not exist"
Sep 10 12:24:32.071672 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.071583    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"]
Sep 10 12:24:32.073616 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:32.073597    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-67fd52b1-7967ffbb5c-mlnnk"]
Sep 10 12:24:33.242619 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:33.242570    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4be91536-3e7c-4846-bba7-7877e3026a80" path="/var/lib/kubelet/pods/4be91536-3e7c-4846-bba7-7877e3026a80/volumes"
Sep 10 12:24:33.243102 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:33.243068    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a1779503-4d10-4f11-8d1f-88fa1ade5f8d" path="/var/lib/kubelet/pods/a1779503-4d10-4f11-8d1f-88fa1ade5f8d/volumes"
Sep 10 12:24:34.864802 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:34.864766    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:24:38.699773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.699733    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"]
Sep 10 12:24:38.700276 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.700250    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4be91536-3e7c-4846-bba7-7877e3026a80" containerName="payload-processing"
Sep 10 12:24:38.700380 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.700280    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4be91536-3e7c-4846-bba7-7877e3026a80" containerName="payload-processing"
Sep 10 12:24:38.700380 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.700316    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a1779503-4d10-4f11-8d1f-88fa1ade5f8d" containerName="revoke-keys"
Sep 10 12:24:38.700380 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.700324    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a1779503-4d10-4f11-8d1f-88fa1ade5f8d" containerName="revoke-keys"
Sep 10 12:24:38.700570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.700406    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4be91536-3e7c-4846-bba7-7877e3026a80" containerName="payload-processing"
Sep 10 12:24:38.700570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.700421    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a1779503-4d10-4f11-8d1f-88fa1ade5f8d" containerName="revoke-keys"
Sep 10 12:24:38.705382 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.705356    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:38.710934 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.710875    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-9e0ca8ed-dockercfg-5tq88\""
Sep 10 12:24:38.711122 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.711105    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-9e0ca8ed\""
Sep 10 12:24:38.720370 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.720339    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"]
Sep 10 12:24:38.832021 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.831988    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtf4w\" (UniqueName: \"kubernetes.io/projected/85c35ed2-8a4b-4609-b171-aef5e35514ea-kube-api-access-qtf4w\") pod \"payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:38.832021 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.832026    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c35ed2-8a4b-4609-b171-aef5e35514ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:38.933417 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.933384    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtf4w\" (UniqueName: \"kubernetes.io/projected/85c35ed2-8a4b-4609-b171-aef5e35514ea-kube-api-access-qtf4w\") pod \"payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:38.933417 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.933420    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c35ed2-8a4b-4609-b171-aef5e35514ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:38.934082 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.934062    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c35ed2-8a4b-4609-b171-aef5e35514ea-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:38.951600 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:38.951545    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtf4w\" (UniqueName: \"kubernetes.io/projected/85c35ed2-8a4b-4609-b171-aef5e35514ea-kube-api-access-qtf4w\") pod \"payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:39.036793 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:39.036766    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:24:39.036974 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:39.036879    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:24:39.050577 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:39.050550    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:39.367474 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:39.367415    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"]
Sep 10 12:24:39.370326 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:24:39.370298    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85c35ed2_8a4b_4609_b171_aef5e35514ea.slice/crio-711c41c80180f35e3e680d2102b5822cfa0c1b7a8a7ef1439b280f92774723b5 WatchSource:0}: Error finding container 711c41c80180f35e3e680d2102b5822cfa0c1b7a8a7ef1439b280f92774723b5: Status 404 returned error can't find the container with id 711c41c80180f35e3e680d2102b5822cfa0c1b7a8a7ef1439b280f92774723b5
Sep 10 12:24:40.092355 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:40.092321    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln" event={"ID":"85c35ed2-8a4b-4609-b171-aef5e35514ea","Type":"ContainerStarted","Data":"7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745"}
Sep 10 12:24:40.092742 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:40.092371    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln" event={"ID":"85c35ed2-8a4b-4609-b171-aef5e35514ea","Type":"ContainerStarted","Data":"711c41c80180f35e3e680d2102b5822cfa0c1b7a8a7ef1439b280f92774723b5"}
Sep 10 12:24:40.092742 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:40.092438    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:45.123790 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:45.123757    2552 generic.go:356] "Generic (PLEG): container finished" podID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerID="064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd" exitCode=0
Sep 10 12:24:45.124339 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:45.123841    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" event={"ID":"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8","Type":"ContainerDied","Data":"064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd"}
Sep 10 12:24:45.124339 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:45.124252    2552 scope.go:117] "RemoveContainer" containerID="064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd"
Sep 10 12:24:45.125345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:45.125318    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:24:45.125406 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:45.125392    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:24:45.125470 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:45.125449    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:24:45.159161 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:45.159100    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln" podStartSLOduration=7.159079813 podStartE2EDuration="7.159079813s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:24:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:24:40.119549098 +0000 UTC m=+891.316064799" watchObservedRunningTime="2026-09-10 12:24:45.159079813 +0000 UTC m=+896.355595515"
Sep 10 12:24:46.131322 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:46.131284    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" event={"ID":"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8","Type":"ContainerStarted","Data":"29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2"}
Sep 10 12:24:47.120159 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:47.120118    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:47.120602 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:47.120571    2552 patch_prober.go:28] interesting pod/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:24:47.120740 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:47.120627    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:24:48.120581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:48.120537    2552 patch_prober.go:28] interesting pod/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:24:48.121756 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:48.121720    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:24:49.119886 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:49.119844    2552 patch_prober.go:28] interesting pod/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:24:49.120579 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:49.120534    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:24:50.120371 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:50.120324    2552 patch_prober.go:28] interesting pod/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:24:50.120909 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:50.120397    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:24:50.588066 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:50.587978    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:24:51.102259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:51.102228    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:24:51.120000 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:51.119969    2552 patch_prober.go:28] interesting pod/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused" start-of-body=
Sep 10 12:24:51.120148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:51.120023    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.42:15021/healthz/ready\": dial tcp 10.134.0.42:15021: connect: connection refused"
Sep 10 12:24:52.123911 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:52.123877    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:52.124281 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:52.124248    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:24:52.124932 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:24:52.124911    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:25:01.032548 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.032509    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"]
Sep 10 12:25:01.037382 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.037364    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"
Sep 10 12:25:01.040091 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.040066    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:25:01.040205 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.040125    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:25:01.047925 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.047904    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"]
Sep 10 12:25:01.137262 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.137224    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eb97e2-900a-4e10-ae17-81c15d291ad1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw\" (UID: \"17eb97e2-900a-4e10-ae17-81c15d291ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"
Sep 10 12:25:01.167688 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.167651    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:25:01.238175 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.238140    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eb97e2-900a-4e10-ae17-81c15d291ad1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw\" (UID: \"17eb97e2-900a-4e10-ae17-81c15d291ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"
Sep 10 12:25:01.238870 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.238851    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eb97e2-900a-4e10-ae17-81c15d291ad1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw\" (UID: \"17eb97e2-900a-4e10-ae17-81c15d291ad1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"
Sep 10 12:25:01.348468 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.348359    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"
Sep 10 12:25:01.648068 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:01.648032    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"]
Sep 10 12:25:01.651024 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:25:01.650995    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17eb97e2_900a_4e10_ae17_81c15d291ad1.slice/crio-e5ea42499a9e3a446e67e8e64f2abb83b6db6ce0dd527b4e31abe1397f31fdec WatchSource:0}: Error finding container e5ea42499a9e3a446e67e8e64f2abb83b6db6ce0dd527b4e31abe1397f31fdec: Status 404 returned error can't find the container with id e5ea42499a9e3a446e67e8e64f2abb83b6db6ce0dd527b4e31abe1397f31fdec
Sep 10 12:25:02.217968 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:02.217930    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw" event={"ID":"17eb97e2-900a-4e10-ae17-81c15d291ad1","Type":"ContainerStarted","Data":"80bb50b2f0d9a7af99cd8ad254c52a0c57a596b8d5449aac09fe9fc97816e935"}
Sep 10 12:25:02.218355 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:02.217987    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw" event={"ID":"17eb97e2-900a-4e10-ae17-81c15d291ad1","Type":"ContainerStarted","Data":"e5ea42499a9e3a446e67e8e64f2abb83b6db6ce0dd527b4e31abe1397f31fdec"}
Sep 10 12:25:03.224771 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:03.224742    2552 generic.go:356] "Generic (PLEG): container finished" podID="17eb97e2-900a-4e10-ae17-81c15d291ad1" containerID="80bb50b2f0d9a7af99cd8ad254c52a0c57a596b8d5449aac09fe9fc97816e935" exitCode=0
Sep 10 12:25:03.224771 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:03.224776    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw" event={"ID":"17eb97e2-900a-4e10-ae17-81c15d291ad1","Type":"ContainerDied","Data":"80bb50b2f0d9a7af99cd8ad254c52a0c57a596b8d5449aac09fe9fc97816e935"}
Sep 10 12:25:04.362064 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:04.362034    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"
Sep 10 12:25:04.470281 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:04.470257    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eb97e2-900a-4e10-ae17-81c15d291ad1-maas-api-service-ca\") pod \"17eb97e2-900a-4e10-ae17-81c15d291ad1\" (UID: \"17eb97e2-900a-4e10-ae17-81c15d291ad1\") "
Sep 10 12:25:04.470605 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:04.470583    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17eb97e2-900a-4e10-ae17-81c15d291ad1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17eb97e2-900a-4e10-ae17-81c15d291ad1" (UID: "17eb97e2-900a-4e10-ae17-81c15d291ad1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:25:04.571062 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:04.571009    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17eb97e2-900a-4e10-ae17-81c15d291ad1-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:25:05.235683 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:05.235659    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"
Sep 10 12:25:05.235835 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:05.235708    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw" event={"ID":"17eb97e2-900a-4e10-ae17-81c15d291ad1","Type":"ContainerDied","Data":"e5ea42499a9e3a446e67e8e64f2abb83b6db6ce0dd527b4e31abe1397f31fdec"}
Sep 10 12:25:05.235835 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:05.235777    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5ea42499a9e3a446e67e8e64f2abb83b6db6ce0dd527b4e31abe1397f31fdec"
Sep 10 12:25:08.621319 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:08.621287    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"]
Sep 10 12:25:08.621880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:08.621524    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln" podUID="85c35ed2-8a4b-4609-b171-aef5e35514ea" containerName="payload-pre-processing" containerID="cri-o://7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745" gracePeriod=30
Sep 10 12:25:08.873611 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:08.873558    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:25:09.009503 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.009476    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtf4w\" (UniqueName: \"kubernetes.io/projected/85c35ed2-8a4b-4609-b171-aef5e35514ea-kube-api-access-qtf4w\") pod \"85c35ed2-8a4b-4609-b171-aef5e35514ea\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") "
Sep 10 12:25:09.009644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.009591    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c35ed2-8a4b-4609-b171-aef5e35514ea-plugins-config-volume\") pod \"85c35ed2-8a4b-4609-b171-aef5e35514ea\" (UID: \"85c35ed2-8a4b-4609-b171-aef5e35514ea\") "
Sep 10 12:25:09.010042 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.010013    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85c35ed2-8a4b-4609-b171-aef5e35514ea-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "85c35ed2-8a4b-4609-b171-aef5e35514ea" (UID: "85c35ed2-8a4b-4609-b171-aef5e35514ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:25:09.011663 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.011640    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85c35ed2-8a4b-4609-b171-aef5e35514ea-kube-api-access-qtf4w" (OuterVolumeSpecName: "kube-api-access-qtf4w") pod "85c35ed2-8a4b-4609-b171-aef5e35514ea" (UID: "85c35ed2-8a4b-4609-b171-aef5e35514ea"). InnerVolumeSpecName "kube-api-access-qtf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:25:09.110597 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.110558    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85c35ed2-8a4b-4609-b171-aef5e35514ea-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:25:09.110597 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.110594    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtf4w\" (UniqueName: \"kubernetes.io/projected/85c35ed2-8a4b-4609-b171-aef5e35514ea-kube-api-access-qtf4w\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:25:09.224397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.224327    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"]
Sep 10 12:25:09.231589 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.231557    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-9e0ca8ed-tvbdw"]
Sep 10 12:25:09.248303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.248275    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17eb97e2-900a-4e10-ae17-81c15d291ad1" path="/var/lib/kubelet/pods/17eb97e2-900a-4e10-ae17-81c15d291ad1/volumes"
Sep 10 12:25:09.255086 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.255059    2552 generic.go:356] "Generic (PLEG): container finished" podID="85c35ed2-8a4b-4609-b171-aef5e35514ea" containerID="7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745" exitCode=0
Sep 10 12:25:09.255209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.255124    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"
Sep 10 12:25:09.255209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.255142    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln" event={"ID":"85c35ed2-8a4b-4609-b171-aef5e35514ea","Type":"ContainerDied","Data":"7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745"}
Sep 10 12:25:09.255209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.255195    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln" event={"ID":"85c35ed2-8a4b-4609-b171-aef5e35514ea","Type":"ContainerDied","Data":"711c41c80180f35e3e680d2102b5822cfa0c1b7a8a7ef1439b280f92774723b5"}
Sep 10 12:25:09.255387 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.255220    2552 scope.go:117] "RemoveContainer" containerID="7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745"
Sep 10 12:25:09.263914 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.263896    2552 scope.go:117] "RemoveContainer" containerID="7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745"
Sep 10 12:25:09.264159 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:25:09.264143    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745\": container with ID starting with 7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745 not found: ID does not exist" containerID="7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745"
Sep 10 12:25:09.264228 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.264164    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745"} err="failed to get container status \"7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745\": rpc error: code = NotFound desc = could not find container \"7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745\": container with ID starting with 7ba5a330b15f4a7531d08c18978e558c8470474b78b21e762ae9d953d0e49745 not found: ID does not exist"
Sep 10 12:25:09.281091 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.281066    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"]
Sep 10 12:25:09.298079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:09.298061    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-9e0ca8ed-67788885f4-k65ln"]
Sep 10 12:25:11.242299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:11.242268    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="85c35ed2-8a4b-4609-b171-aef5e35514ea" path="/var/lib/kubelet/pods/85c35ed2-8a4b-4609-b171-aef5e35514ea/volumes"
Sep 10 12:25:15.457475 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:15.457444    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:25:22.808608 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.808578    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"]
Sep 10 12:25:22.808989 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.808915    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17eb97e2-900a-4e10-ae17-81c15d291ad1" containerName="revoke-keys"
Sep 10 12:25:22.808989 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.808926    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="17eb97e2-900a-4e10-ae17-81c15d291ad1" containerName="revoke-keys"
Sep 10 12:25:22.808989 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.808944    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="85c35ed2-8a4b-4609-b171-aef5e35514ea" containerName="payload-pre-processing"
Sep 10 12:25:22.808989 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.808949    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="85c35ed2-8a4b-4609-b171-aef5e35514ea" containerName="payload-pre-processing"
Sep 10 12:25:22.809123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.809004    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="17eb97e2-900a-4e10-ae17-81c15d291ad1" containerName="revoke-keys"
Sep 10 12:25:22.809123 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.809014    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="85c35ed2-8a4b-4609-b171-aef5e35514ea" containerName="payload-pre-processing"
Sep 10 12:25:22.812224 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.812207    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:22.817723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.817704    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fb35b9\"/\"default-dockercfg-jdw7v\""
Sep 10 12:25:22.818976 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.818953    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fb35b9\"/\"openshift-service-ca.crt\""
Sep 10 12:25:22.818976 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.818968    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-fb35b9\"/\"test-model-fb35b9-kserve-self-signed-certs\""
Sep 10 12:25:22.819122 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.819078    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-fb35b9\"/\"kube-root-ca.crt\""
Sep 10 12:25:22.832128 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.832109    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"]
Sep 10 12:25:22.934761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.934732    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k56k8\" (UniqueName: \"kubernetes.io/projected/c12e92f2-6928-41b1-8a97-5f51b98d61b8-kube-api-access-k56k8\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:22.934884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.934828    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-home\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:22.934884 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.934851    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tls-certs\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:22.934959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.934895    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-model-cache\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:22.934959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.934913    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tmp-dir\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:22.934959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:22.934928    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-dshm\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036086 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036061    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-home\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036092    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tls-certs\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036119    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-model-cache\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036243    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tmp-dir\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036288    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-dshm\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036421 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036322    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k56k8\" (UniqueName: \"kubernetes.io/projected/c12e92f2-6928-41b1-8a97-5f51b98d61b8-kube-api-access-k56k8\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036459    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-home\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036566 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036506    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-model-cache\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.036626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.036571    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tmp-dir\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.038526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.038510    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-dshm\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.038797 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.038773    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tls-certs\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.048109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.048089    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k56k8\" (UniqueName: \"kubernetes.io/projected/c12e92f2-6928-41b1-8a97-5f51b98d61b8-kube-api-access-k56k8\") pod \"test-model-fb35b9-kserve-86bbdd7cbc-ktfls\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") " pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.121285 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.121241    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:23.405466 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:23.405442    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"]
Sep 10 12:25:23.408167 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:25:23.408142    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc12e92f2_6928_41b1_8a97_5f51b98d61b8.slice/crio-20041ed57141cfb8c255326a26c994b5c19eeba377247437b272dc2fbb744f23 WatchSource:0}: Error finding container 20041ed57141cfb8c255326a26c994b5c19eeba377247437b272dc2fbb744f23: Status 404 returned error can't find the container with id 20041ed57141cfb8c255326a26c994b5c19eeba377247437b272dc2fbb744f23
Sep 10 12:25:24.333517 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:24.333485    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls" event={"ID":"c12e92f2-6928-41b1-8a97-5f51b98d61b8","Type":"ContainerStarted","Data":"499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c"}
Sep 10 12:25:24.334036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:24.333525    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls" event={"ID":"c12e92f2-6928-41b1-8a97-5f51b98d61b8","Type":"ContainerStarted","Data":"20041ed57141cfb8c255326a26c994b5c19eeba377247437b272dc2fbb744f23"}
Sep 10 12:25:24.378891 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:24.378846    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls" podStartSLOduration=2.37883157 podStartE2EDuration="2.37883157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:25:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:25:24.377402226 +0000 UTC m=+935.573917925" watchObservedRunningTime="2026-09-10 12:25:24.37883157 +0000 UTC m=+935.575347270"
Sep 10 12:25:28.526855 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:28.526825    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:25:33.122485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:33.122413    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:33.122894 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:33.122504    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:33.135012 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:33.134977    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:33.386556 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:33.386469    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:25:42.960359 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:42.960322    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:25:57.185347 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:25:57.185314    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:26:01.514388 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.514360    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"]
Sep 10 12:26:01.514837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.514579    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s" podUID="6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" containerName="payload-processing" containerID="cri-o://c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e" gracePeriod=30
Sep 10 12:26:01.545579 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.545553    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"]
Sep 10 12:26:01.545810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.545774    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr" podUID="3e490442-3318-4958-91de-e30c13ed4f01" containerName="payload-pre-processing" containerID="cri-o://b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b" gracePeriod=30
Sep 10 12:26:01.820954 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.820929    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:26:01.823657 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.823638    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:26:01.968231 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.968196    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-plugins-config-volume\") pod \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") "
Sep 10 12:26:01.968417 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.968291    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m4j8v\" (UniqueName: \"kubernetes.io/projected/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-kube-api-access-m4j8v\") pod \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\" (UID: \"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c\") "
Sep 10 12:26:01.968417 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.968343    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e490442-3318-4958-91de-e30c13ed4f01-plugins-config-volume\") pod \"3e490442-3318-4958-91de-e30c13ed4f01\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") "
Sep 10 12:26:01.968543 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.968418    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-b9xdh\" (UniqueName: \"kubernetes.io/projected/3e490442-3318-4958-91de-e30c13ed4f01-kube-api-access-b9xdh\") pod \"3e490442-3318-4958-91de-e30c13ed4f01\" (UID: \"3e490442-3318-4958-91de-e30c13ed4f01\") "
Sep 10 12:26:01.968739 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.968712    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" (UID: "6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:26:01.968824 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.968742    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3e490442-3318-4958-91de-e30c13ed4f01-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3e490442-3318-4958-91de-e30c13ed4f01" (UID: "3e490442-3318-4958-91de-e30c13ed4f01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:26:01.970653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.970625    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-kube-api-access-m4j8v" (OuterVolumeSpecName: "kube-api-access-m4j8v") pod "6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" (UID: "6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c"). InnerVolumeSpecName "kube-api-access-m4j8v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:01.970757 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:01.970700    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e490442-3318-4958-91de-e30c13ed4f01-kube-api-access-b9xdh" (OuterVolumeSpecName: "kube-api-access-b9xdh") pod "3e490442-3318-4958-91de-e30c13ed4f01" (UID: "3e490442-3318-4958-91de-e30c13ed4f01"). InnerVolumeSpecName "kube-api-access-b9xdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:02.069448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.069319    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m4j8v\" (UniqueName: \"kubernetes.io/projected/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-kube-api-access-m4j8v\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:02.069448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.069367    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3e490442-3318-4958-91de-e30c13ed4f01-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:02.069448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.069376    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b9xdh\" (UniqueName: \"kubernetes.io/projected/3e490442-3318-4958-91de-e30c13ed4f01-kube-api-access-b9xdh\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:02.069448 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.069385    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:02.510734 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.510698    2552 generic.go:356] "Generic (PLEG): container finished" podID="6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" containerID="c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e" exitCode=0
Sep 10 12:26:02.510931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.510763    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"
Sep 10 12:26:02.510931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.510779    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s" event={"ID":"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c","Type":"ContainerDied","Data":"c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e"}
Sep 10 12:26:02.510931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.510829    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s" event={"ID":"6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c","Type":"ContainerDied","Data":"4fafa27d1ce2898af1199dad80ee5cf3b8372062546ee478cb390a3a0037f8b7"}
Sep 10 12:26:02.510931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.510841    2552 scope.go:117] "RemoveContainer" containerID="c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e"
Sep 10 12:26:02.512219 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.512190    2552 generic.go:356] "Generic (PLEG): container finished" podID="3e490442-3318-4958-91de-e30c13ed4f01" containerID="b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b" exitCode=0
Sep 10 12:26:02.512349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.512235    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"
Sep 10 12:26:02.512349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.512292    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr" event={"ID":"3e490442-3318-4958-91de-e30c13ed4f01","Type":"ContainerDied","Data":"b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b"}
Sep 10 12:26:02.512349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.512340    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr" event={"ID":"3e490442-3318-4958-91de-e30c13ed4f01","Type":"ContainerDied","Data":"61b3efbd7bc80419c4949845cb38f55c433b4390a3096f95e9c1b41f4ffaf0b4"}
Sep 10 12:26:02.520598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.520167    2552 scope.go:117] "RemoveContainer" containerID="c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e"
Sep 10 12:26:02.520598 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:02.520463    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e\": container with ID starting with c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e not found: ID does not exist" containerID="c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e"
Sep 10 12:26:02.520598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.520486    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e"} err="failed to get container status \"c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e\": rpc error: code = NotFound desc = could not find container \"c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e\": container with ID starting with c71b3507ad54bebed5a2c5d4586aac944fe95ef919314f53e974b157ac9fe83e not found: ID does not exist"
Sep 10 12:26:02.520598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.520502    2552 scope.go:117] "RemoveContainer" containerID="b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b"
Sep 10 12:26:02.533654 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.533637    2552 scope.go:117] "RemoveContainer" containerID="b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b"
Sep 10 12:26:02.533900 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:02.533881    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b\": container with ID starting with b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b not found: ID does not exist" containerID="b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b"
Sep 10 12:26:02.533957 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.533912    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b"} err="failed to get container status \"b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b\": rpc error: code = NotFound desc = could not find container \"b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b\": container with ID starting with b1746bfb62c0b97d7c5dac6f10cc909c2de2f27c9ef8454e96029ede3a4d8f4b not found: ID does not exist"
Sep 10 12:26:02.549207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.549186    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"]
Sep 10 12:26:02.558566 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.558545    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-f615cc-67d6bcc77d-ch9tr"]
Sep 10 12:26:02.571727 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.571704    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"]
Sep 10 12:26:02.578688 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:02.578670    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-explicit-f615cc-598988fd5c-jhn8s"]
Sep 10 12:26:03.242156 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:03.242124    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e490442-3318-4958-91de-e30c13ed4f01" path="/var/lib/kubelet/pods/3e490442-3318-4958-91de-e30c13ed4f01/volumes"
Sep 10 12:26:03.242498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:03.242485    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" path="/var/lib/kubelet/pods/6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c/volumes"
Sep 10 12:26:07.578347 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:07.578315    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:26:09.879118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:09.879086    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"]
Sep 10 12:26:09.879508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:09.879298    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy" containerID="cri-o://29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2" gracePeriod=30
Sep 10 12:26:10.635116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.635046    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"]
Sep 10 12:26:10.635463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.635447    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" containerName="payload-processing"
Sep 10 12:26:10.635526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.635465    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" containerName="payload-processing"
Sep 10 12:26:10.635526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.635484    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e490442-3318-4958-91de-e30c13ed4f01" containerName="payload-pre-processing"
Sep 10 12:26:10.635526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.635491    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e490442-3318-4958-91de-e30c13ed4f01" containerName="payload-pre-processing"
Sep 10 12:26:10.635628 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.635545    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="6d8755b4-2d5e-4fdc-b0d6-1663c3ff959c" containerName="payload-processing"
Sep 10 12:26:10.635628 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.635556    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e490442-3318-4958-91de-e30c13ed4f01" containerName="payload-pre-processing"
Sep 10 12:26:10.639704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.639688    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.642057 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.642036    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-1d8e56-openshift-default-dockercfg-86vr5\""
Sep 10 12:26:10.655752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.655729    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"]
Sep 10 12:26:10.743261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743226    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-credential-socket\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743273    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-data\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743338    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-token\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743417    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-socket\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743474    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rd2tj\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-kube-api-access-rd2tj\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743662 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743542    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istiod-ca-cert\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743662 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743575    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-certs\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743662 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743602    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-envoy\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.743790 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.743662    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-podinfo\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844464    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-certs\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844497 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844503    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-envoy\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844539    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-podinfo\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844583    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-credential-socket\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844612    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-data\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844731    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-token\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844867    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-certs\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844891    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-socket\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.844975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844932    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rd2tj\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-kube-api-access-rd2tj\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.845141 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.844976    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-credential-socket\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.845141 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.845025    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-data\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.845141 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.845052    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istiod-ca-cert\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.845141 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.845112    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-socket\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.845947 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.845924    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istiod-ca-cert\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.847100 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.847077    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-envoy\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.847221 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.847203    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-podinfo\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.879012 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.878978    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-token\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.879914 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.879892    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rd2tj\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-kube-api-access-rd2tj\") pod \"e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") " pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:10.949885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:10.949795    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:11.235632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.235606    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"]
Sep 10 12:26:11.238367 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:26:11.238338    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69f8f0d4_0e5e_4d7a_9b96_6c2dfb98b985.slice/crio-8c48e7b7c2f9077e2c839536a58841bf9b3469ab169d9301fadf07c0b86812db WatchSource:0}: Error finding container 8c48e7b7c2f9077e2c839536a58841bf9b3469ab169d9301fadf07c0b86812db: Status 404 returned error can't find the container with id 8c48e7b7c2f9077e2c839536a58841bf9b3469ab169d9301fadf07c0b86812db
Sep 10 12:26:11.241176 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.241142    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:26:11.241263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.241213    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:26:11.241263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.241245    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:26:11.559067 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.558984    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c" event={"ID":"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985","Type":"ContainerStarted","Data":"ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa"}
Sep 10 12:26:11.559067 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.559041    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c" event={"ID":"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985","Type":"ContainerStarted","Data":"8c48e7b7c2f9077e2c839536a58841bf9b3469ab169d9301fadf07c0b86812db"}
Sep 10 12:26:11.592178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.592125    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c" podStartSLOduration=1.592110049 podStartE2EDuration="1.592110049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:26:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:26:11.589725139 +0000 UTC m=+982.786240839" watchObservedRunningTime="2026-09-10 12:26:11.592110049 +0000 UTC m=+982.788625749"
Sep 10 12:26:11.950829 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:11.950798    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:12.955609 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:12.955582    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:13.568447 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:13.568394    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:13.569473 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:13.569452    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:15.136058 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.136036    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:26:15.289374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289301    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-token\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289340    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-podinfo\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289386    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-credential-socket\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289625 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289569    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-socket\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289691 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289633    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-envoy\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289691 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289658    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istiod-ca-cert\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289691 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289679    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-data\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289842 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289712    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-certs\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.289842 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:15.289842 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289752    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-whpc2\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-kube-api-access-whpc2\") pod \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\" (UID: \"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8\") "
Sep 10 12:26:15.290027 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.290003    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:15.290092 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.289987    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-data" (OuterVolumeSpecName: "istio-data") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:15.290092 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.290052    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:15.290278 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.290260    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.290325 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.290285    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.290325 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.290299    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.290325 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.290312    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.290693 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.290620    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:26:15.291745 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.291722    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-token" (OuterVolumeSpecName: "istio-token") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:15.291745 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.291740    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:26:15.291953 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.291935    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:15.292134 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.292118    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-kube-api-access-whpc2" (OuterVolumeSpecName: "kube-api-access-whpc2") pod "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" (UID: "4f209ffd-75c4-4c14-b404-0e5df8f3e7c8"). InnerVolumeSpecName "kube-api-access-whpc2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:15.390988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.390940    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.390988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.390979    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.390988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.390988    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whpc2\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-kube-api-access-whpc2\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.390988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.391000    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.391302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.391015    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:15.580328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.580236    2552 generic.go:356] "Generic (PLEG): container finished" podID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerID="29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2" exitCode=0
Sep 10 12:26:15.580328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.580285    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"
Sep 10 12:26:15.580328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.580317    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" event={"ID":"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8","Type":"ContainerDied","Data":"29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2"}
Sep 10 12:26:15.580657 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.580370    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d" event={"ID":"4f209ffd-75c4-4c14-b404-0e5df8f3e7c8","Type":"ContainerDied","Data":"eed1839be5bde7e0b59a6b64a73cdcc382e2be6dae29406676071c256b1dfe8d"}
Sep 10 12:26:15.580657 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.580392    2552 scope.go:117] "RemoveContainer" containerID="29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2"
Sep 10 12:26:15.589871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.589855    2552 scope.go:117] "RemoveContainer" containerID="064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd"
Sep 10 12:26:15.599837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.599815    2552 scope.go:117] "RemoveContainer" containerID="29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2"
Sep 10 12:26:15.600271 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:15.600252    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2\": container with ID starting with 29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2 not found: ID does not exist" containerID="29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2"
Sep 10 12:26:15.600321 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.600279    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2"} err="failed to get container status \"29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2\": rpc error: code = NotFound desc = could not find container \"29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2\": container with ID starting with 29f9f93d5d229915fe313d21b00a207e41e1c8783471c921d479707968e396f2 not found: ID does not exist"
Sep 10 12:26:15.600321 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.600295    2552 scope.go:117] "RemoveContainer" containerID="064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd"
Sep 10 12:26:15.600593 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:15.600569    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd\": container with ID starting with 064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd not found: ID does not exist" containerID="064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd"
Sep 10 12:26:15.600697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.600598    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd"} err="failed to get container status \"064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd\": rpc error: code = NotFound desc = could not find container \"064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd\": container with ID starting with 064f8e8843a69523d1c67e0b4268b6416be4aa12bba95669e1b15cd006275bcd not found: ID does not exist"
Sep 10 12:26:15.605191 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.605169    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"]
Sep 10 12:26:15.610003 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:15.609983    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-f615cc-openshift-default-67987d7bc6-mlk9d"]
Sep 10 12:26:17.242412 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:17.242379    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" path="/var/lib/kubelet/pods/4f209ffd-75c4-4c14-b404-0e5df8f3e7c8/volumes"
Sep 10 12:26:24.041788 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:24.041756    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"]
Sep 10 12:26:24.042291 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:24.041999    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c" podUID="69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" containerName="istio-proxy" containerID="cri-o://ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa" gracePeriod=30
Sep 10 12:26:28.571218 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:28.571177    2552 patch_prober.go:28] interesting pod/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 12:26:28.571607 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:28.571248    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c" podUID="69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 12:26:29.289402 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.289381    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:29.317316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317215    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-podinfo\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317261    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-token\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317293    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istiod-ca-cert\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317367    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-socket\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317392    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rd2tj\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-kube-api-access-rd2tj\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317415    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-envoy\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317471    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-credential-socket\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317522    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-data\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317603 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317578    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-certs\") pod \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\" (UID: \"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985\") "
Sep 10 12:26:29.317895 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317676    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:29.317895 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317766    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:26:29.318005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.317913    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:29.318174 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.318087    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:29.318355 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.318319    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-data" (OuterVolumeSpecName: "istio-data") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:29.318685 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.318658    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.318685 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.318689    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.318862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.318702    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.318862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.318714    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.318862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.318728    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.320374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.320347    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-token" (OuterVolumeSpecName: "istio-token") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:29.320525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.320382    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-kube-api-access-rd2tj" (OuterVolumeSpecName: "kube-api-access-rd2tj") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "kube-api-access-rd2tj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:29.320662 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.320640    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:26:29.320724 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.320699    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" (UID: "69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:29.420055 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.420024    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.420055 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.420051    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.420055 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.420059    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rd2tj\" (UniqueName: \"kubernetes.io/projected/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-kube-api-access-rd2tj\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.420300 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.420070    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:29.649393 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.649358    2552 generic.go:356] "Generic (PLEG): container finished" podID="69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" containerID="ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa" exitCode=0
Sep 10 12:26:29.649835 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.649407    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"
Sep 10 12:26:29.649835 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.649467    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c" event={"ID":"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985","Type":"ContainerDied","Data":"ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa"}
Sep 10 12:26:29.649835 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.649521    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c" event={"ID":"69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985","Type":"ContainerDied","Data":"8c48e7b7c2f9077e2c839536a58841bf9b3469ab169d9301fadf07c0b86812db"}
Sep 10 12:26:29.649835 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.649544    2552 scope.go:117] "RemoveContainer" containerID="ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa"
Sep 10 12:26:29.658752 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.658732    2552 scope.go:117] "RemoveContainer" containerID="ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa"
Sep 10 12:26:29.659005 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:29.658983    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa\": container with ID starting with ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa not found: ID does not exist" containerID="ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa"
Sep 10 12:26:29.659068 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.659018    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa"} err="failed to get container status \"ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa\": rpc error: code = NotFound desc = could not find container \"ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa\": container with ID starting with ac4f46937c36ff0090b166f275e6b8d73018586559761f1cefbab4f6a8f25eaa not found: ID does not exist"
Sep 10 12:26:29.673658 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.673632    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"]
Sep 10 12:26:29.687188 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:29.683551    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-1d8e56-openshift-default-7499fb7fd6-4zr2c"]
Sep 10 12:26:31.241856 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:31.241824    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" path="/var/lib/kubelet/pods/69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985/volumes"
Sep 10 12:26:39.061404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:39.061372    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:26:42.110573 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:42.110538    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"]
Sep 10 12:26:43.059279 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.059242    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"]
Sep 10 12:26:43.229552 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.229502    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"]
Sep 10 12:26:43.230008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.229897    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy"
Sep 10 12:26:43.230008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.229910    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy"
Sep 10 12:26:43.230008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.229925    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy"
Sep 10 12:26:43.230008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.229930    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy"
Sep 10 12:26:43.230008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.229938    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" containerName="istio-proxy"
Sep 10 12:26:43.230008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.229943    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" containerName="istio-proxy"
Sep 10 12:26:43.230008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.230008    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="69f8f0d4-0e5e-4d7a-9b96-6c2dfb98b985" containerName="istio-proxy"
Sep 10 12:26:43.230228 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.230015    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy"
Sep 10 12:26:43.234551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.234526    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"
Sep 10 12:26:43.238345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.238325    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:26:43.238650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.238630    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:26:43.244671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.244648    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"]
Sep 10 12:26:43.342077 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.341967    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c4bbaa-0621-4468-baa4-342b297d1268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2\" (UID: \"c8c4bbaa-0621-4468-baa4-342b297d1268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"
Sep 10 12:26:43.443447 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.443386    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c4bbaa-0621-4468-baa4-342b297d1268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2\" (UID: \"c8c4bbaa-0621-4468-baa4-342b297d1268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"
Sep 10 12:26:43.444038 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.444013    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c4bbaa-0621-4468-baa4-342b297d1268-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2\" (UID: \"c8c4bbaa-0621-4468-baa4-342b297d1268\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"
Sep 10 12:26:43.547550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.547504    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"
Sep 10 12:26:43.829882 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:43.829853    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"]
Sep 10 12:26:43.831808 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:26:43.831784    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc8c4bbaa_0621_4468_baa4_342b297d1268.slice/crio-938cd70959c50f9623c17aec6fe0265a1b9d99f033b1d621a0d36fee370ada76 WatchSource:0}: Error finding container 938cd70959c50f9623c17aec6fe0265a1b9d99f033b1d621a0d36fee370ada76: Status 404 returned error can't find the container with id 938cd70959c50f9623c17aec6fe0265a1b9d99f033b1d621a0d36fee370ada76
Sep 10 12:26:44.723723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:44.723639    2552 generic.go:356] "Generic (PLEG): container finished" podID="c8c4bbaa-0621-4468-baa4-342b297d1268" containerID="fc380757a5e3c33d1793d8a08a31183bf4e19031d0bbdd0c9a5e1b95f281f656" exitCode=0
Sep 10 12:26:44.724074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:44.723730    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2" event={"ID":"c8c4bbaa-0621-4468-baa4-342b297d1268","Type":"ContainerDied","Data":"fc380757a5e3c33d1793d8a08a31183bf4e19031d0bbdd0c9a5e1b95f281f656"}
Sep 10 12:26:44.724074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:44.723789    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2" event={"ID":"c8c4bbaa-0621-4468-baa4-342b297d1268","Type":"ContainerStarted","Data":"938cd70959c50f9623c17aec6fe0265a1b9d99f033b1d621a0d36fee370ada76"}
Sep 10 12:26:45.856670 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:45.856645    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"
Sep 10 12:26:45.965050 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:45.965017    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c4bbaa-0621-4468-baa4-342b297d1268-maas-api-service-ca\") pod \"c8c4bbaa-0621-4468-baa4-342b297d1268\" (UID: \"c8c4bbaa-0621-4468-baa4-342b297d1268\") "
Sep 10 12:26:45.965382 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:45.965361    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c8c4bbaa-0621-4468-baa4-342b297d1268-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c8c4bbaa-0621-4468-baa4-342b297d1268" (UID: "c8c4bbaa-0621-4468-baa4-342b297d1268"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:26:46.066570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:46.066488    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c8c4bbaa-0621-4468-baa4-342b297d1268-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:46.733828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:46.733790    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2" event={"ID":"c8c4bbaa-0621-4468-baa4-342b297d1268","Type":"ContainerDied","Data":"938cd70959c50f9623c17aec6fe0265a1b9d99f033b1d621a0d36fee370ada76"}
Sep 10 12:26:46.733828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:46.733828    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"
Sep 10 12:26:46.734083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:46.733838    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="938cd70959c50f9623c17aec6fe0265a1b9d99f033b1d621a0d36fee370ada76"
Sep 10 12:26:52.435761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.435730    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:26:52.437269 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.437248    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"]
Sep 10 12:26:52.437602 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.437590    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c8c4bbaa-0621-4468-baa4-342b297d1268" containerName="revoke-keys"
Sep 10 12:26:52.437651 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.437605    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c8c4bbaa-0621-4468-baa4-342b297d1268" containerName="revoke-keys"
Sep 10 12:26:52.437691 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.437673    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="4f209ffd-75c4-4c14-b404-0e5df8f3e7c8" containerName="istio-proxy"
Sep 10 12:26:52.437691 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.437685    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c8c4bbaa-0621-4468-baa4-342b297d1268" containerName="revoke-keys"
Sep 10 12:26:52.441777 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.441760    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.444695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.444679    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-062bb857-openshift-default-dockercfg-29m58\""
Sep 10 12:26:52.444695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.444663    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 10 12:26:52.465729 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.465707    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"]
Sep 10 12:26:52.519384 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519360    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-envoy\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.519576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519409    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mpz27\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-kube-api-access-mpz27\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.519576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519506    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-socket\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.519576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519549    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-credential-socket\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.519732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519688    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istiod-ca-cert\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.519808 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519787    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-certs\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.519917 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519897    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-podinfo\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.519977 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519924    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-token\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.520041 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.519987    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-data\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621198 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621176    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istiod-ca-cert\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621238    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-certs\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621268    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-podinfo\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621290    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-token\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621338    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-data\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621392    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-envoy\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621454    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mpz27\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-kube-api-access-mpz27\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621486    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-socket\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621515    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-credential-socket\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621802 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621694    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-certs\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621864 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621844    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-data\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621920 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621873    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-credential-socket\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.621987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.621928    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istiod-ca-cert\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.622198 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.622003    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-socket\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.623836 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.623817    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-envoy\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.623924 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.623891    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-podinfo\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.633405 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.633321    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-token\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.634880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.634860    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mpz27\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-kube-api-access-mpz27\") pod \"e2e-mt-062bb857-openshift-default-664d988556-6t7qg\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") " pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:52.751298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:52.751236    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:53.037277 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:53.037252    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"]
Sep 10 12:26:53.038511 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:26:53.038482    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3d26f41_c799_43d7_ad2c_3a1cd6e3d18d.slice/crio-7f31bf807f0dc66d381c68d504b7deadd1c35405387257e17b2f3e940cdc0d59 WatchSource:0}: Error finding container 7f31bf807f0dc66d381c68d504b7deadd1c35405387257e17b2f3e940cdc0d59: Status 404 returned error can't find the container with id 7f31bf807f0dc66d381c68d504b7deadd1c35405387257e17b2f3e940cdc0d59
Sep 10 12:26:53.040598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:53.040566    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:26:53.040672 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:53.040631    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:26:53.040672 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:53.040660    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:26:53.768697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:53.768664    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" event={"ID":"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d","Type":"ContainerStarted","Data":"d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4"}
Sep 10 12:26:53.769044 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:53.768712    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" event={"ID":"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d","Type":"ContainerStarted","Data":"7f31bf807f0dc66d381c68d504b7deadd1c35405387257e17b2f3e940cdc0d59"}
Sep 10 12:26:53.803323 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:53.803269    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podStartSLOduration=1.8032531729999999 podStartE2EDuration="1.803253173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:26:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:26:53.80306534 +0000 UTC m=+1024.999581041" watchObservedRunningTime="2026-09-10 12:26:53.803253173 +0000 UTC m=+1024.999768873"
Sep 10 12:26:54.069570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.069496    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"]
Sep 10 12:26:54.069710 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.069677    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd" podUID="ad35c8a9-bda9-4033-9bff-93e5b6329bf7" containerName="payload-pre-processing" containerID="cri-o://8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74" gracePeriod=30
Sep 10 12:26:54.324193 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.324141    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:26:54.438133 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.438108    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-plugins-config-volume\") pod \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") "
Sep 10 12:26:54.438272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.438181    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hpdvp\" (UniqueName: \"kubernetes.io/projected/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-kube-api-access-hpdvp\") pod \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\" (UID: \"ad35c8a9-bda9-4033-9bff-93e5b6329bf7\") "
Sep 10 12:26:54.438507 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.438468    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ad35c8a9-bda9-4033-9bff-93e5b6329bf7" (UID: "ad35c8a9-bda9-4033-9bff-93e5b6329bf7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:26:54.440301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.440281    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-kube-api-access-hpdvp" (OuterVolumeSpecName: "kube-api-access-hpdvp") pod "ad35c8a9-bda9-4033-9bff-93e5b6329bf7" (UID: "ad35c8a9-bda9-4033-9bff-93e5b6329bf7"). InnerVolumeSpecName "kube-api-access-hpdvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:54.503052 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.503020    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"]
Sep 10 12:26:54.512044 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.512014    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-7576a1-2r5t2"]
Sep 10 12:26:54.539296 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.539271    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:54.539296 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.539293    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hpdvp\" (UniqueName: \"kubernetes.io/projected/ad35c8a9-bda9-4033-9bff-93e5b6329bf7-kube-api-access-hpdvp\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:54.751582 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.751554    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:54.756442 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.756403    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:54.774384 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.774354    2552 generic.go:356] "Generic (PLEG): container finished" podID="ad35c8a9-bda9-4033-9bff-93e5b6329bf7" containerID="8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74" exitCode=0
Sep 10 12:26:54.774726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.774402    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"
Sep 10 12:26:54.774726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.774457    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd" event={"ID":"ad35c8a9-bda9-4033-9bff-93e5b6329bf7","Type":"ContainerDied","Data":"8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74"}
Sep 10 12:26:54.774726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.774502    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd" event={"ID":"ad35c8a9-bda9-4033-9bff-93e5b6329bf7","Type":"ContainerDied","Data":"a87c47764ee8ddcd6803103dac7607980f2111ee87c9d03f54c1efe1bba9c6bf"}
Sep 10 12:26:54.774726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.774524    2552 scope.go:117] "RemoveContainer" containerID="8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74"
Sep 10 12:26:54.775005 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.774755    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:54.775632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.775611    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:26:54.783281 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.783267    2552 scope.go:117] "RemoveContainer" containerID="8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74"
Sep 10 12:26:54.783627 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:54.783607    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74\": container with ID starting with 8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74 not found: ID does not exist" containerID="8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74"
Sep 10 12:26:54.783715 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.783634    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74"} err="failed to get container status \"8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74\": rpc error: code = NotFound desc = could not find container \"8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74\": container with ID starting with 8ad8de74f072034924e20d09e5116aa41534511ec82b5004af5fd481dd324a74 not found: ID does not exist"
Sep 10 12:26:54.839808 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.839785    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"]
Sep 10 12:26:54.860591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:54.860572    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-7576a1-7f6675bc54-48rcd"]
Sep 10 12:26:55.242383 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:55.242355    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ad35c8a9-bda9-4033-9bff-93e5b6329bf7" path="/var/lib/kubelet/pods/ad35c8a9-bda9-4033-9bff-93e5b6329bf7/volumes"
Sep 10 12:26:55.242733 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:55.242716    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c8c4bbaa-0621-4468-baa4-342b297d1268" path="/var/lib/kubelet/pods/c8c4bbaa-0621-4468-baa4-342b297d1268/volumes"
Sep 10 12:26:57.133066 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.133022    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4" podUID="2588855a-b69f-44fb-9ab7-45fafa86a83e" containerName="main" containerID="cri-o://359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e" gracePeriod=45
Sep 10 12:26:57.385300 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.385246    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:26:57.463178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463149    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-home\") pod \"2588855a-b69f-44fb-9ab7-45fafa86a83e\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") "
Sep 10 12:26:57.463312 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463202    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-dshm\") pod \"2588855a-b69f-44fb-9ab7-45fafa86a83e\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") "
Sep 10 12:26:57.463312 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463226    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-model-cache\") pod \"2588855a-b69f-44fb-9ab7-45fafa86a83e\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") "
Sep 10 12:26:57.463312 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463283    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2588855a-b69f-44fb-9ab7-45fafa86a83e-tls-certs\") pod \"2588855a-b69f-44fb-9ab7-45fafa86a83e\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") "
Sep 10 12:26:57.463312 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463301    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-45hhp\" (UniqueName: \"kubernetes.io/projected/2588855a-b69f-44fb-9ab7-45fafa86a83e-kube-api-access-45hhp\") pod \"2588855a-b69f-44fb-9ab7-45fafa86a83e\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") "
Sep 10 12:26:57.463584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463344    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-tmp-dir\") pod \"2588855a-b69f-44fb-9ab7-45fafa86a83e\" (UID: \"2588855a-b69f-44fb-9ab7-45fafa86a83e\") "
Sep 10 12:26:57.463584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463394    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-home" (OuterVolumeSpecName: "home") pod "2588855a-b69f-44fb-9ab7-45fafa86a83e" (UID: "2588855a-b69f-44fb-9ab7-45fafa86a83e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:57.463584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463515    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-model-cache" (OuterVolumeSpecName: "model-cache") pod "2588855a-b69f-44fb-9ab7-45fafa86a83e" (UID: "2588855a-b69f-44fb-9ab7-45fafa86a83e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:57.463769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463667    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2588855a-b69f-44fb-9ab7-45fafa86a83e" (UID: "2588855a-b69f-44fb-9ab7-45fafa86a83e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:57.463769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463729    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:57.463769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.463747    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:57.465667 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.465636    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2588855a-b69f-44fb-9ab7-45fafa86a83e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2588855a-b69f-44fb-9ab7-45fafa86a83e" (UID: "2588855a-b69f-44fb-9ab7-45fafa86a83e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:26:57.465772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.465714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-dshm" (OuterVolumeSpecName: "dshm") pod "2588855a-b69f-44fb-9ab7-45fafa86a83e" (UID: "2588855a-b69f-44fb-9ab7-45fafa86a83e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:57.465772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.465714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2588855a-b69f-44fb-9ab7-45fafa86a83e-kube-api-access-45hhp" (OuterVolumeSpecName: "kube-api-access-45hhp") pod "2588855a-b69f-44fb-9ab7-45fafa86a83e" (UID: "2588855a-b69f-44fb-9ab7-45fafa86a83e"). InnerVolumeSpecName "kube-api-access-45hhp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:57.564532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.564504    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:57.564532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.564527    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2588855a-b69f-44fb-9ab7-45fafa86a83e-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:57.564688 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.564538    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-45hhp\" (UniqueName: \"kubernetes.io/projected/2588855a-b69f-44fb-9ab7-45fafa86a83e-kube-api-access-45hhp\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:57.564688 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.564548    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2588855a-b69f-44fb-9ab7-45fafa86a83e-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:57.792668 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.792583    2552 generic.go:356] "Generic (PLEG): container finished" podID="2588855a-b69f-44fb-9ab7-45fafa86a83e" containerID="359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e" exitCode=0
Sep 10 12:26:57.792668 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.792623    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"
Sep 10 12:26:57.792865 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.792713    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4" event={"ID":"2588855a-b69f-44fb-9ab7-45fafa86a83e","Type":"ContainerDied","Data":"359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e"}
Sep 10 12:26:57.792865 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.792753    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4" event={"ID":"2588855a-b69f-44fb-9ab7-45fafa86a83e","Type":"ContainerDied","Data":"b5e091f8cd06f623a744c278d8acf9099c6bcb452eca123ec4c843fa5b34e59b"}
Sep 10 12:26:57.792865 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.792771    2552 scope.go:117] "RemoveContainer" containerID="359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e"
Sep 10 12:26:57.806804 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.806787    2552 scope.go:117] "RemoveContainer" containerID="359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e"
Sep 10 12:26:57.807084 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:57.807066    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e\": container with ID starting with 359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e not found: ID does not exist" containerID="359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e"
Sep 10 12:26:57.807138 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.807093    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e"} err="failed to get container status \"359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e\": rpc error: code = NotFound desc = could not find container \"359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e\": container with ID starting with 359a74d02532ec37167bb2cc749ca165e355c9285f3f8e07cdff0862c119c29e not found: ID does not exist"
Sep 10 12:26:57.832316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.832294    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"]
Sep 10 12:26:57.846809 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:57.846783    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-7576a1/test-model-7576a1-kserve-9f9ffd955-vxhf4"]
Sep 10 12:26:58.078731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.078668    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls" podUID="c12e92f2-6928-41b1-8a97-5f51b98d61b8" containerName="main" containerID="cri-o://499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c" gracePeriod=45
Sep 10 12:26:58.331547 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.331488    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:26:58.471325 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471296    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-home\") pod \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") "
Sep 10 12:26:58.471325 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471325    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-model-cache\") pod \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") "
Sep 10 12:26:58.471581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471354    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-dshm\") pod \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") "
Sep 10 12:26:58.471581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471382    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k56k8\" (UniqueName: \"kubernetes.io/projected/c12e92f2-6928-41b1-8a97-5f51b98d61b8-kube-api-access-k56k8\") pod \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") "
Sep 10 12:26:58.471581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471439    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tmp-dir\") pod \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") "
Sep 10 12:26:58.471749 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471586    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-home" (OuterVolumeSpecName: "home") pod "c12e92f2-6928-41b1-8a97-5f51b98d61b8" (UID: "c12e92f2-6928-41b1-8a97-5f51b98d61b8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:58.471749 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471594    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tls-certs\") pod \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\" (UID: \"c12e92f2-6928-41b1-8a97-5f51b98d61b8\") "
Sep 10 12:26:58.471749 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471610    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-model-cache" (OuterVolumeSpecName: "model-cache") pod "c12e92f2-6928-41b1-8a97-5f51b98d61b8" (UID: "c12e92f2-6928-41b1-8a97-5f51b98d61b8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:58.471749 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471685    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c12e92f2-6928-41b1-8a97-5f51b98d61b8" (UID: "c12e92f2-6928-41b1-8a97-5f51b98d61b8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:58.471954 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471935    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:58.472024 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471961    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:58.472024 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.471977    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:58.473843 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.473822    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-dshm" (OuterVolumeSpecName: "dshm") pod "c12e92f2-6928-41b1-8a97-5f51b98d61b8" (UID: "c12e92f2-6928-41b1-8a97-5f51b98d61b8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:26:58.474098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.474081    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c12e92f2-6928-41b1-8a97-5f51b98d61b8" (UID: "c12e92f2-6928-41b1-8a97-5f51b98d61b8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:26:58.474880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.474850    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c12e92f2-6928-41b1-8a97-5f51b98d61b8-kube-api-access-k56k8" (OuterVolumeSpecName: "kube-api-access-k56k8") pod "c12e92f2-6928-41b1-8a97-5f51b98d61b8" (UID: "c12e92f2-6928-41b1-8a97-5f51b98d61b8"). InnerVolumeSpecName "kube-api-access-k56k8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:26:58.572370 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.572347    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c12e92f2-6928-41b1-8a97-5f51b98d61b8-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:58.572370 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.572370    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c12e92f2-6928-41b1-8a97-5f51b98d61b8-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:58.572517 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.572379    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k56k8\" (UniqueName: \"kubernetes.io/projected/c12e92f2-6928-41b1-8a97-5f51b98d61b8-kube-api-access-k56k8\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:26:58.646636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.646606    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"]
Sep 10 12:26:58.646921 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.646909    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ad35c8a9-bda9-4033-9bff-93e5b6329bf7" containerName="payload-pre-processing"
Sep 10 12:26:58.646978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.646923    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ad35c8a9-bda9-4033-9bff-93e5b6329bf7" containerName="payload-pre-processing"
Sep 10 12:26:58.646978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.646944    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2588855a-b69f-44fb-9ab7-45fafa86a83e" containerName="main"
Sep 10 12:26:58.646978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.646949    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2588855a-b69f-44fb-9ab7-45fafa86a83e" containerName="main"
Sep 10 12:26:58.646978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.646958    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c12e92f2-6928-41b1-8a97-5f51b98d61b8" containerName="main"
Sep 10 12:26:58.646978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.646963    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c12e92f2-6928-41b1-8a97-5f51b98d61b8" containerName="main"
Sep 10 12:26:58.647147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.647010    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ad35c8a9-bda9-4033-9bff-93e5b6329bf7" containerName="payload-pre-processing"
Sep 10 12:26:58.647147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.647019    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2588855a-b69f-44fb-9ab7-45fafa86a83e" containerName="main"
Sep 10 12:26:58.647147 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.647027    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c12e92f2-6928-41b1-8a97-5f51b98d61b8" containerName="main"
Sep 10 12:26:58.649969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.649955    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.658181 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.658162    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-062bb857-dockercfg-dv2bf\""
Sep 10 12:26:58.658282 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.658239    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-062bb857\""
Sep 10 12:26:58.666994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.666974    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"]
Sep 10 12:26:58.737270 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.737241    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"]
Sep 10 12:26:58.740626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.740607    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:58.754463 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.754421    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"]
Sep 10 12:26:58.774297 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.774273    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-glz5v\" (UniqueName: \"kubernetes.io/projected/44498763-5500-4a40-a0a1-6878e2a0a940-kube-api-access-glz5v\") pod \"payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.774413 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.774394    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44498763-5500-4a40-a0a1-6878e2a0a940-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.799708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.799679    2552 generic.go:356] "Generic (PLEG): container finished" podID="c12e92f2-6928-41b1-8a97-5f51b98d61b8" containerID="499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c" exitCode=0
Sep 10 12:26:58.799828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.799715    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"
Sep 10 12:26:58.799828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.799715    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls" event={"ID":"c12e92f2-6928-41b1-8a97-5f51b98d61b8","Type":"ContainerDied","Data":"499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c"}
Sep 10 12:26:58.799943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.799839    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls" event={"ID":"c12e92f2-6928-41b1-8a97-5f51b98d61b8","Type":"ContainerDied","Data":"20041ed57141cfb8c255326a26c994b5c19eeba377247437b272dc2fbb744f23"}
Sep 10 12:26:58.799943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.799857    2552 scope.go:117] "RemoveContainer" containerID="499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c"
Sep 10 12:26:58.808488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.808467    2552 scope.go:117] "RemoveContainer" containerID="499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c"
Sep 10 12:26:58.808784 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:26:58.808759    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c\": container with ID starting with 499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c not found: ID does not exist" containerID="499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c"
Sep 10 12:26:58.808840 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.808796    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c"} err="failed to get container status \"499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c\": rpc error: code = NotFound desc = could not find container \"499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c\": container with ID starting with 499500cbcfcfd0a203f3d1a624d680eabd7acb493f1ccb0a9bb78ecb360f8f0c not found: ID does not exist"
Sep 10 12:26:58.823213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.823190    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"]
Sep 10 12:26:58.828353 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.828333    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-fb35b9/test-model-fb35b9-kserve-86bbdd7cbc-ktfls"]
Sep 10 12:26:58.875640 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.875621    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44498763-5500-4a40-a0a1-6878e2a0a940-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.875737 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.875653    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-glz5v\" (UniqueName: \"kubernetes.io/projected/44498763-5500-4a40-a0a1-6878e2a0a940-kube-api-access-glz5v\") pod \"payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.875737 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.875711    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dgm2g\" (UniqueName: \"kubernetes.io/projected/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-kube-api-access-dgm2g\") pod \"payload-processing-e2e-mt-062bb857-8558476949-wxsd6\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") " pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:58.875737 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.875730    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-062bb857-8558476949-wxsd6\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") " pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:58.876313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.876296    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44498763-5500-4a40-a0a1-6878e2a0a940-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.886718 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.886702    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-glz5v\" (UniqueName: \"kubernetes.io/projected/44498763-5500-4a40-a0a1-6878e2a0a940-kube-api-access-glz5v\") pod \"payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.959820 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.959763    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:58.976072 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.976045    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dgm2g\" (UniqueName: \"kubernetes.io/projected/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-kube-api-access-dgm2g\") pod \"payload-processing-e2e-mt-062bb857-8558476949-wxsd6\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") " pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:58.976180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.976080    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-062bb857-8558476949-wxsd6\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") " pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:58.976764 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.976743    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-plugins-config-volume\") pod \"payload-processing-e2e-mt-062bb857-8558476949-wxsd6\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") " pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:58.986233 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:58.986205    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dgm2g\" (UniqueName: \"kubernetes.io/projected/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-kube-api-access-dgm2g\") pod \"payload-processing-e2e-mt-062bb857-8558476949-wxsd6\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") " pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:59.050319 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.050286    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:59.236095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.235992    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"]
Sep 10 12:26:59.238661 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:26:59.238636    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44498763_5500_4a40_a0a1_6878e2a0a940.slice/crio-9ec2790e8163576a53c8bd62db700776897491db18ec774dd19a9d0b3bcd04d3 WatchSource:0}: Error finding container 9ec2790e8163576a53c8bd62db700776897491db18ec774dd19a9d0b3bcd04d3: Status 404 returned error can't find the container with id 9ec2790e8163576a53c8bd62db700776897491db18ec774dd19a9d0b3bcd04d3
Sep 10 12:26:59.242292 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.242268    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2588855a-b69f-44fb-9ab7-45fafa86a83e" path="/var/lib/kubelet/pods/2588855a-b69f-44fb-9ab7-45fafa86a83e/volumes"
Sep 10 12:26:59.242695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.242680    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c12e92f2-6928-41b1-8a97-5f51b98d61b8" path="/var/lib/kubelet/pods/c12e92f2-6928-41b1-8a97-5f51b98d61b8/volumes"
Sep 10 12:26:59.328561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.328537    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"]
Sep 10 12:26:59.331138 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:26:59.331108    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf6b9fcf9_9009_4b94_a4c4_270cf81550ee.slice/crio-90084af805be5e44a041e3dcb800e65bcabf888d36633173225524a249baca7f WatchSource:0}: Error finding container 90084af805be5e44a041e3dcb800e65bcabf888d36633173225524a249baca7f: Status 404 returned error can't find the container with id 90084af805be5e44a041e3dcb800e65bcabf888d36633173225524a249baca7f
Sep 10 12:26:59.808498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.808461    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6" event={"ID":"f6b9fcf9-9009-4b94-a4c4-270cf81550ee","Type":"ContainerStarted","Data":"06b19db3f947f410e21a9083ee125f6a0e1318d0018f7676c1f7a5a17a65bfb1"}
Sep 10 12:26:59.808987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.808596    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:26:59.808987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.808631    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6" event={"ID":"f6b9fcf9-9009-4b94-a4c4-270cf81550ee","Type":"ContainerStarted","Data":"90084af805be5e44a041e3dcb800e65bcabf888d36633173225524a249baca7f"}
Sep 10 12:26:59.810317 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.810277    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl" event={"ID":"44498763-5500-4a40-a0a1-6878e2a0a940","Type":"ContainerStarted","Data":"eaf54169f1390b0c629c71fa671344646d05de4c285e33138026d8423bf67d9b"}
Sep 10 12:26:59.810491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.810329    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl" event={"ID":"44498763-5500-4a40-a0a1-6878e2a0a940","Type":"ContainerStarted","Data":"9ec2790e8163576a53c8bd62db700776897491db18ec774dd19a9d0b3bcd04d3"}
Sep 10 12:26:59.810491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.810359    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:26:59.835962 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.835913    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6" podStartSLOduration=1.8358989650000002 podStartE2EDuration="1.835898965s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:26:59.834669432 +0000 UTC m=+1031.031185136" watchObservedRunningTime="2026-09-10 12:26:59.835898965 +0000 UTC m=+1031.032414665"
Sep 10 12:26:59.858599 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:26:59.858555    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl" podStartSLOduration=1.858539927 podStartE2EDuration="1.858539927s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:26:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:26:59.858109053 +0000 UTC m=+1031.054624755" watchObservedRunningTime="2026-09-10 12:26:59.858539927 +0000 UTC m=+1031.055055649"
Sep 10 12:27:01.956229 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:01.956193    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:27:02.066243 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.066207    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"]
Sep 10 12:27:02.069593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.069575    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"
Sep 10 12:27:02.072536 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.072515    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:27:02.072650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.072558    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:27:02.081495 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.081474    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"]
Sep 10 12:27:02.203921 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.203893    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ff3e87-a9e2-417a-82d0-d1fe45788962-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg\" (UID: \"47ff3e87-a9e2-417a-82d0-d1fe45788962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"
Sep 10 12:27:02.304864 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.304774    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ff3e87-a9e2-417a-82d0-d1fe45788962-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg\" (UID: \"47ff3e87-a9e2-417a-82d0-d1fe45788962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"
Sep 10 12:27:02.305421 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.305396    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ff3e87-a9e2-417a-82d0-d1fe45788962-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg\" (UID: \"47ff3e87-a9e2-417a-82d0-d1fe45788962\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"
Sep 10 12:27:02.379947 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.379902    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"
Sep 10 12:27:02.654934 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.654904    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"]
Sep 10 12:27:02.655943 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:27:02.655910    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47ff3e87_a9e2_417a_82d0_d1fe45788962.slice/crio-f2db037c9559560a3524118ab6dc068c05b3735b49a522494d5db01cad619d75 WatchSource:0}: Error finding container f2db037c9559560a3524118ab6dc068c05b3735b49a522494d5db01cad619d75: Status 404 returned error can't find the container with id f2db037c9559560a3524118ab6dc068c05b3735b49a522494d5db01cad619d75
Sep 10 12:27:02.830299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:02.830268    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg" event={"ID":"47ff3e87-a9e2-417a-82d0-d1fe45788962","Type":"ContainerStarted","Data":"f2db037c9559560a3524118ab6dc068c05b3735b49a522494d5db01cad619d75"}
Sep 10 12:27:03.836246 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:03.836210    2552 generic.go:356] "Generic (PLEG): container finished" podID="47ff3e87-a9e2-417a-82d0-d1fe45788962" containerID="5b0ad20320394c3698cd673eb49b8419a0c7f53c98655bf4b3c21a0cb3c58de0" exitCode=0
Sep 10 12:27:03.836662 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:03.836292    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg" event={"ID":"47ff3e87-a9e2-417a-82d0-d1fe45788962","Type":"ContainerDied","Data":"5b0ad20320394c3698cd673eb49b8419a0c7f53c98655bf4b3c21a0cb3c58de0"}
Sep 10 12:27:04.983534 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:04.983508    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"
Sep 10 12:27:05.131746 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:05.131710    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ff3e87-a9e2-417a-82d0-d1fe45788962-maas-api-service-ca\") pod \"47ff3e87-a9e2-417a-82d0-d1fe45788962\" (UID: \"47ff3e87-a9e2-417a-82d0-d1fe45788962\") "
Sep 10 12:27:05.132135 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:05.132108    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47ff3e87-a9e2-417a-82d0-d1fe45788962-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "47ff3e87-a9e2-417a-82d0-d1fe45788962" (UID: "47ff3e87-a9e2-417a-82d0-d1fe45788962"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:27:05.232802 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:05.232764    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/47ff3e87-a9e2-417a-82d0-d1fe45788962-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:05.847644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:05.847614    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"
Sep 10 12:27:05.847644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:05.847637    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg" event={"ID":"47ff3e87-a9e2-417a-82d0-d1fe45788962","Type":"ContainerDied","Data":"f2db037c9559560a3524118ab6dc068c05b3735b49a522494d5db01cad619d75"}
Sep 10 12:27:05.847823 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:05.847684    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2db037c9559560a3524118ab6dc068c05b3735b49a522494d5db01cad619d75"
Sep 10 12:27:10.817594 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:10.817555    2552 patch_prober.go:28] interesting pod/e2e-mt-062bb857-openshift-default-664d988556-6t7qg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 10 12:27:10.818065 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:10.817619    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 10 12:27:10.819172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:10.819149    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:27:10.819311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:10.819265    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:27:12.676797 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.676765    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"]
Sep 10 12:27:12.677281 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.676984    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn" podUID="80703ce5-2296-4b40-95bd-ad104c73189c" containerName="payload-pre-processing" containerID="cri-o://529a1bbbc59a8bf241f84d9ebe7e81e5892abc98f339cbf715efbc72d6f6f5e2" gracePeriod=30
Sep 10 12:27:12.870150 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.870116    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"]
Sep 10 12:27:12.873763 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.873735    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-fb35b9-5k9lg"]
Sep 10 12:27:12.890910 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.890874    2552 generic.go:356] "Generic (PLEG): container finished" podID="80703ce5-2296-4b40-95bd-ad104c73189c" containerID="529a1bbbc59a8bf241f84d9ebe7e81e5892abc98f339cbf715efbc72d6f6f5e2" exitCode=0
Sep 10 12:27:12.891062 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.890950    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn" event={"ID":"80703ce5-2296-4b40-95bd-ad104c73189c","Type":"ContainerDied","Data":"529a1bbbc59a8bf241f84d9ebe7e81e5892abc98f339cbf715efbc72d6f6f5e2"}
Sep 10 12:27:12.893122 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.893098    2552 generic.go:356] "Generic (PLEG): container finished" podID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerID="d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4" exitCode=0
Sep 10 12:27:12.893242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.893162    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" event={"ID":"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d","Type":"ContainerDied","Data":"d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4"}
Sep 10 12:27:12.893584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.893569    2552 scope.go:117] "RemoveContainer" containerID="d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4"
Sep 10 12:27:12.894762 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.894715    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:27:12.894845 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.894825    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:27:12.894900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.894883    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:27:12.946374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.946317    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:27:12.998871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:12.998841    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f9bpf\" (UniqueName: \"kubernetes.io/projected/80703ce5-2296-4b40-95bd-ad104c73189c-kube-api-access-f9bpf\") pod \"80703ce5-2296-4b40-95bd-ad104c73189c\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") "
Sep 10 12:27:13.001302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.001270    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80703ce5-2296-4b40-95bd-ad104c73189c-kube-api-access-f9bpf" (OuterVolumeSpecName: "kube-api-access-f9bpf") pod "80703ce5-2296-4b40-95bd-ad104c73189c" (UID: "80703ce5-2296-4b40-95bd-ad104c73189c"). InnerVolumeSpecName "kube-api-access-f9bpf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:27:13.099819 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.099776    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80703ce5-2296-4b40-95bd-ad104c73189c-plugins-config-volume\") pod \"80703ce5-2296-4b40-95bd-ad104c73189c\" (UID: \"80703ce5-2296-4b40-95bd-ad104c73189c\") "
Sep 10 12:27:13.100036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.100020    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9bpf\" (UniqueName: \"kubernetes.io/projected/80703ce5-2296-4b40-95bd-ad104c73189c-kube-api-access-f9bpf\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:13.100234 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.100203    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80703ce5-2296-4b40-95bd-ad104c73189c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "80703ce5-2296-4b40-95bd-ad104c73189c" (UID: "80703ce5-2296-4b40-95bd-ad104c73189c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:27:13.200572 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.200485    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80703ce5-2296-4b40-95bd-ad104c73189c-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:13.241959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.241927    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="47ff3e87-a9e2-417a-82d0-d1fe45788962" path="/var/lib/kubelet/pods/47ff3e87-a9e2-417a-82d0-d1fe45788962/volumes"
Sep 10 12:27:13.902516 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.901636    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn" event={"ID":"80703ce5-2296-4b40-95bd-ad104c73189c","Type":"ContainerDied","Data":"3fb304bd164bc082d12c77d9610fc6a5208cbed2db179d460b31e75efbc72803"}
Sep 10 12:27:13.902516 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.901715    2552 scope.go:117] "RemoveContainer" containerID="529a1bbbc59a8bf241f84d9ebe7e81e5892abc98f339cbf715efbc72d6f6f5e2"
Sep 10 12:27:13.902516 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.901893    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"
Sep 10 12:27:13.908484 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.907658    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" event={"ID":"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d","Type":"ContainerStarted","Data":"201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55"}
Sep 10 12:27:13.955812 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.955779    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"]
Sep 10 12:27:13.959615 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:13.959583    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fb35b9-8459b944fd-v8crn"]
Sep 10 12:27:14.751450 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:14.751403    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:27:14.751679 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:14.751657    2552 patch_prober.go:28] interesting pod/e2e-mt-062bb857-openshift-default-664d988556-6t7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 12:27:14.751726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:14.751708    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 12:27:15.242447 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:15.242385    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80703ce5-2296-4b40-95bd-ad104c73189c" path="/var/lib/kubelet/pods/80703ce5-2296-4b40-95bd-ad104c73189c/volumes"
Sep 10 12:27:15.354179 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:15.354134    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:27:15.751893 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:15.751851    2552 patch_prober.go:28] interesting pod/e2e-mt-062bb857-openshift-default-664d988556-6t7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 12:27:15.752080 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:15.751926    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 12:27:16.752073 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:16.752027    2552 patch_prober.go:28] interesting pod/e2e-mt-062bb857-openshift-default-664d988556-6t7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 12:27:16.752523 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:16.752086    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 12:27:17.752509 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:17.752386    2552 patch_prober.go:28] interesting pod/e2e-mt-062bb857-openshift-default-664d988556-6t7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 12:27:17.753193 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:17.752554    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 12:27:18.752261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:18.752214    2552 patch_prober.go:28] interesting pod/e2e-mt-062bb857-openshift-default-664d988556-6t7qg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused" start-of-body=
Sep 10 12:27:18.752477 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:18.752292    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.51:15021/healthz/ready\": dial tcp 10.134.0.51:15021: connect: connection refused"
Sep 10 12:27:19.756183 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:19.756149    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:27:19.756588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:19.756441    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:27:19.757262 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:19.757243    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:27:30.100330 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.100283    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:27:30.127643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.127606    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"]
Sep 10 12:27:30.127990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.127977    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="47ff3e87-a9e2-417a-82d0-d1fe45788962" containerName="revoke-keys"
Sep 10 12:27:30.128036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.127992    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="47ff3e87-a9e2-417a-82d0-d1fe45788962" containerName="revoke-keys"
Sep 10 12:27:30.128036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.128006    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80703ce5-2296-4b40-95bd-ad104c73189c" containerName="payload-pre-processing"
Sep 10 12:27:30.128036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.128011    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="80703ce5-2296-4b40-95bd-ad104c73189c" containerName="payload-pre-processing"
Sep 10 12:27:30.128148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.128069    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="47ff3e87-a9e2-417a-82d0-d1fe45788962" containerName="revoke-keys"
Sep 10 12:27:30.128148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.128082    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="80703ce5-2296-4b40-95bd-ad104c73189c" containerName="payload-pre-processing"
Sep 10 12:27:30.131383 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.131359    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"
Sep 10 12:27:30.134602 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.134570    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:27:30.135213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.135195    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:27:30.147957 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.147929    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"]
Sep 10 12:27:30.244371 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.244330    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-062bb857-v4hnd\" (UID: \"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"
Sep 10 12:27:30.345897 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.345846    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-062bb857-v4hnd\" (UID: \"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"
Sep 10 12:27:30.346522 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.346503    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-062bb857-v4hnd\" (UID: \"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"
Sep 10 12:27:30.441594 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.441558    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"
Sep 10 12:27:30.720722 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.720491    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"]
Sep 10 12:27:30.723585 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:27:30.723554    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2324a00_d0c8_4e2a_bb3d_feaa86f8c72c.slice/crio-a21cf3e89794f51b45218ad0217e222560d391f9727b4e8f9ea9f9e6f3fb951e WatchSource:0}: Error finding container a21cf3e89794f51b45218ad0217e222560d391f9727b4e8f9ea9f9e6f3fb951e: Status 404 returned error can't find the container with id a21cf3e89794f51b45218ad0217e222560d391f9727b4e8f9ea9f9e6f3fb951e
Sep 10 12:27:30.996740 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:30.996703    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd" event={"ID":"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c","Type":"ContainerStarted","Data":"a21cf3e89794f51b45218ad0217e222560d391f9727b4e8f9ea9f9e6f3fb951e"}
Sep 10 12:27:32.006296 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:32.006261    2552 generic.go:356] "Generic (PLEG): container finished" podID="b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c" containerID="69227f7b72d74341fd99c45328b571b9e3cb2cd3aefe02dc5e19279fd02b9dbc" exitCode=0
Sep 10 12:27:32.006780 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:32.006383    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd" event={"ID":"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c","Type":"ContainerDied","Data":"69227f7b72d74341fd99c45328b571b9e3cb2cd3aefe02dc5e19279fd02b9dbc"}
Sep 10 12:27:33.139619 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:33.139595    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"
Sep 10 12:27:33.175744 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:33.175716    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c-maas-api-service-ca\") pod \"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c\" (UID: \"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c\") "
Sep 10 12:27:33.176048 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:33.176025    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c" (UID: "b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:27:33.276725 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:33.276662    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:34.016505 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:34.016474    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"
Sep 10 12:27:34.016505 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:34.016485    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd" event={"ID":"b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c","Type":"ContainerDied","Data":"a21cf3e89794f51b45218ad0217e222560d391f9727b4e8f9ea9f9e6f3fb951e"}
Sep 10 12:27:34.016714 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:34.016533    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a21cf3e89794f51b45218ad0217e222560d391f9727b4e8f9ea9f9e6f3fb951e"
Sep 10 12:27:40.844704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:40.844617    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"]
Sep 10 12:27:40.845150 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:40.844883    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6" podUID="f6b9fcf9-9009-4b94-a4c4-270cf81550ee" containerName="payload-processing" containerID="cri-o://06b19db3f947f410e21a9083ee125f6a0e1318d0018f7676c1f7a5a17a65bfb1" gracePeriod=30
Sep 10 12:27:40.860688 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:40.860663    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"]
Sep 10 12:27:40.860961 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:40.860848    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl" podUID="44498763-5500-4a40-a0a1-6878e2a0a940" containerName="payload-pre-processing" containerID="cri-o://eaf54169f1390b0c629c71fa671344646d05de4c285e33138026d8423bf67d9b" gracePeriod=30
Sep 10 12:27:41.056961 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.056106    2552 generic.go:356] "Generic (PLEG): container finished" podID="f6b9fcf9-9009-4b94-a4c4-270cf81550ee" containerID="06b19db3f947f410e21a9083ee125f6a0e1318d0018f7676c1f7a5a17a65bfb1" exitCode=0
Sep 10 12:27:41.056961 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.056242    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6" event={"ID":"f6b9fcf9-9009-4b94-a4c4-270cf81550ee","Type":"ContainerDied","Data":"06b19db3f947f410e21a9083ee125f6a0e1318d0018f7676c1f7a5a17a65bfb1"}
Sep 10 12:27:41.058703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.058684    2552 generic.go:356] "Generic (PLEG): container finished" podID="44498763-5500-4a40-a0a1-6878e2a0a940" containerID="eaf54169f1390b0c629c71fa671344646d05de4c285e33138026d8423bf67d9b" exitCode=0
Sep 10 12:27:41.058807 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.058732    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl" event={"ID":"44498763-5500-4a40-a0a1-6878e2a0a940","Type":"ContainerDied","Data":"eaf54169f1390b0c629c71fa671344646d05de4c285e33138026d8423bf67d9b"}
Sep 10 12:27:41.089254 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.089228    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"]
Sep 10 12:27:41.094838 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.094784    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-062bb857-v4hnd"]
Sep 10 12:27:41.135981 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.135963    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:27:41.138098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.138072    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:27:41.242576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.242549    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c" path="/var/lib/kubelet/pods/b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c/volumes"
Sep 10 12:27:41.245636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.245612    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44498763-5500-4a40-a0a1-6878e2a0a940-plugins-config-volume\") pod \"44498763-5500-4a40-a0a1-6878e2a0a940\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") "
Sep 10 12:27:41.245785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.245688    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-glz5v\" (UniqueName: \"kubernetes.io/projected/44498763-5500-4a40-a0a1-6878e2a0a940-kube-api-access-glz5v\") pod \"44498763-5500-4a40-a0a1-6878e2a0a940\" (UID: \"44498763-5500-4a40-a0a1-6878e2a0a940\") "
Sep 10 12:27:41.245785 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.245747    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dgm2g\" (UniqueName: \"kubernetes.io/projected/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-kube-api-access-dgm2g\") pod \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") "
Sep 10 12:27:41.245912 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.245859    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-plugins-config-volume\") pod \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\" (UID: \"f6b9fcf9-9009-4b94-a4c4-270cf81550ee\") "
Sep 10 12:27:41.246042 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.246014    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44498763-5500-4a40-a0a1-6878e2a0a940-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "44498763-5500-4a40-a0a1-6878e2a0a940" (UID: "44498763-5500-4a40-a0a1-6878e2a0a940"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:27:41.246229 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.246206    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f6b9fcf9-9009-4b94-a4c4-270cf81550ee" (UID: "f6b9fcf9-9009-4b94-a4c4-270cf81550ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:27:41.247828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.247806    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44498763-5500-4a40-a0a1-6878e2a0a940-kube-api-access-glz5v" (OuterVolumeSpecName: "kube-api-access-glz5v") pod "44498763-5500-4a40-a0a1-6878e2a0a940" (UID: "44498763-5500-4a40-a0a1-6878e2a0a940"). InnerVolumeSpecName "kube-api-access-glz5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:27:41.247904 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.247834    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-kube-api-access-dgm2g" (OuterVolumeSpecName: "kube-api-access-dgm2g") pod "f6b9fcf9-9009-4b94-a4c4-270cf81550ee" (UID: "f6b9fcf9-9009-4b94-a4c4-270cf81550ee"). InnerVolumeSpecName "kube-api-access-dgm2g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:27:41.346822 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.346771    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dgm2g\" (UniqueName: \"kubernetes.io/projected/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-kube-api-access-dgm2g\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:41.346822 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.346792    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f6b9fcf9-9009-4b94-a4c4-270cf81550ee-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:41.346822 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.346802    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44498763-5500-4a40-a0a1-6878e2a0a940-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:41.346822 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:41.346811    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-glz5v\" (UniqueName: \"kubernetes.io/projected/44498763-5500-4a40-a0a1-6878e2a0a940-kube-api-access-glz5v\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:42.071480 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.071450    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"
Sep 10 12:27:42.071480 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.071456    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6" event={"ID":"f6b9fcf9-9009-4b94-a4c4-270cf81550ee","Type":"ContainerDied","Data":"90084af805be5e44a041e3dcb800e65bcabf888d36633173225524a249baca7f"}
Sep 10 12:27:42.071945 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.071551    2552 scope.go:117] "RemoveContainer" containerID="06b19db3f947f410e21a9083ee125f6a0e1318d0018f7676c1f7a5a17a65bfb1"
Sep 10 12:27:42.073073 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.073051    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"
Sep 10 12:27:42.073169 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.073066    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl" event={"ID":"44498763-5500-4a40-a0a1-6878e2a0a940","Type":"ContainerDied","Data":"9ec2790e8163576a53c8bd62db700776897491db18ec774dd19a9d0b3bcd04d3"}
Sep 10 12:27:42.080835 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.080817    2552 scope.go:117] "RemoveContainer" containerID="eaf54169f1390b0c629c71fa671344646d05de4c285e33138026d8423bf67d9b"
Sep 10 12:27:42.101547 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.101515    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"]
Sep 10 12:27:42.104554 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.104533    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-062bb857-dd6c48c74-8rkfl"]
Sep 10 12:27:42.118906 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.118883    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"]
Sep 10 12:27:42.126669 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:42.126631    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-062bb857-8558476949-wxsd6"]
Sep 10 12:27:43.242008 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:43.241968    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="44498763-5500-4a40-a0a1-6878e2a0a940" path="/var/lib/kubelet/pods/44498763-5500-4a40-a0a1-6878e2a0a940/volumes"
Sep 10 12:27:43.242418 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:43.242322    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f6b9fcf9-9009-4b94-a4c4-270cf81550ee" path="/var/lib/kubelet/pods/f6b9fcf9-9009-4b94-a4c4-270cf81550ee/volumes"
Sep 10 12:27:43.703178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:43.703152    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:27:48.652521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:48.652477    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"]
Sep 10 12:27:48.653028 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:48.652833    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" containerID="cri-o://201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55" gracePeriod=30
Sep 10 12:27:49.759388 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:49.759352    2552 patch_prober.go:28] interesting pod/e2e-mt-062bb857-openshift-default-664d988556-6t7qg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 10 12:27:49.759791 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:49.759442    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 12:27:50.366936 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.366907    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"]
Sep 10 12:27:50.367239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367228    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c" containerName="revoke-keys"
Sep 10 12:27:50.367298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367242    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c" containerName="revoke-keys"
Sep 10 12:27:50.367298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367261    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="44498763-5500-4a40-a0a1-6878e2a0a940" containerName="payload-pre-processing"
Sep 10 12:27:50.367298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367267    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="44498763-5500-4a40-a0a1-6878e2a0a940" containerName="payload-pre-processing"
Sep 10 12:27:50.367298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367276    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f6b9fcf9-9009-4b94-a4c4-270cf81550ee" containerName="payload-processing"
Sep 10 12:27:50.367298 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367281    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f6b9fcf9-9009-4b94-a4c4-270cf81550ee" containerName="payload-processing"
Sep 10 12:27:50.367470 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367331    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f6b9fcf9-9009-4b94-a4c4-270cf81550ee" containerName="payload-processing"
Sep 10 12:27:50.367470 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367340    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="44498763-5500-4a40-a0a1-6878e2a0a940" containerName="payload-pre-processing"
Sep 10 12:27:50.367470 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.367346    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b2324a00-d0c8-4e2a-bb3d-feaa86f8c72c" containerName="revoke-keys"
Sep 10 12:27:50.371562 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.371547    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.375085 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.375064    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-f89a79-openshift-default-dockercfg-462mr\""
Sep 10 12:27:50.397843 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.397814    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"]
Sep 10 12:27:50.521628 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521595    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-podinfo\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.521806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521635    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3e800d-e06e-4251-8dc7-77573950ade0-istiod-ca-cert\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.521806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521654    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jfjd\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-kube-api-access-7jfjd\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.521806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521717    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-envoy\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.521806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521756    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-credential-socket\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.521806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521776    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-token\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.521806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521804    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-certs\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.522042 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521854    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-data\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.522042 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.521892    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-socket\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.622754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.622659    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-socket\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.622754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.622707    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-podinfo\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.622754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.622732    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3e800d-e06e-4251-8dc7-77573950ade0-istiod-ca-cert\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623022 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.622847    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jfjd\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-kube-api-access-7jfjd\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623022 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.622913    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-envoy\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623022 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.622957    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-credential-socket\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623022 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.622981    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-token\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623286 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.623023    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-certs\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623286 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.623078    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-data\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623286 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.623126    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-socket\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623566 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.623542    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-credential-socket\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623661 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.623589    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-data\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623717 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.623675    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3e800d-e06e-4251-8dc7-77573950ade0-istiod-ca-cert\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.623760 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.623734    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-certs\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.625323 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.625302    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-envoy\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.625410 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.625394    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-podinfo\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.633021 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.632996    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jfjd\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-kube-api-access-7jfjd\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.635798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.635777    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-token\") pod \"e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") " pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.681398 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.681373    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:50.983700 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.983558    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"]
Sep 10 12:27:50.985766 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:27:50.985738    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b3e800d_e06e_4251_8dc7_77573950ade0.slice/crio-69b08a2242e31d2a28a2888e5bcb410563bcfd4bc3ebe88634dbeaf19c68f30e WatchSource:0}: Error finding container 69b08a2242e31d2a28a2888e5bcb410563bcfd4bc3ebe88634dbeaf19c68f30e: Status 404 returned error can't find the container with id 69b08a2242e31d2a28a2888e5bcb410563bcfd4bc3ebe88634dbeaf19c68f30e
Sep 10 12:27:50.987867 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.987834    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:27:50.987937 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.987900    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:27:50.987937 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:50.987927    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:27:51.120645 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:51.120612    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" event={"ID":"0b3e800d-e06e-4251-8dc7-77573950ade0","Type":"ContainerStarted","Data":"62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644"}
Sep 10 12:27:51.120803 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:51.120676    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" event={"ID":"0b3e800d-e06e-4251-8dc7-77573950ade0","Type":"ContainerStarted","Data":"69b08a2242e31d2a28a2888e5bcb410563bcfd4bc3ebe88634dbeaf19c68f30e"}
Sep 10 12:27:51.151053 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:51.150990    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podStartSLOduration=1.150967808 podStartE2EDuration="1.150967808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:27:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:27:51.149355561 +0000 UTC m=+1082.345871262" watchObservedRunningTime="2026-09-10 12:27:51.150967808 +0000 UTC m=+1082.347483509"
Sep 10 12:27:51.682135 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:51.682104    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:52.686769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:52.686730    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:53.135682 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:53.135656    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:53.136718 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:53.136695    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:27:53.900131 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:53.900094    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:27:54.054987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.054891    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-data\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.054987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.054963    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-socket\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.054987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.054994    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-token\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.055253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055056    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-envoy\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.055253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055079    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-credential-socket\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.055253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055101    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-certs\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.055253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055137    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istiod-ca-cert\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.055253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055228    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-data" (OuterVolumeSpecName: "istio-data") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:27:54.055550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055289    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:27:54.055550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055308    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-podinfo\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.055550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055389    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mpz27\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-kube-api-access-mpz27\") pod \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\" (UID: \"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d\") "
Sep 10 12:27:54.055550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055455    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:27:54.055550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055470    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:27:54.055810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055557    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:27:54.055810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055707    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.055810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055721    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.055810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055732    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.055810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055743    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.055810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.055752    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.057411 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.057385    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:27:54.057594 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.057576    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-kube-api-access-mpz27" (OuterVolumeSpecName: "kube-api-access-mpz27") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "kube-api-access-mpz27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:27:54.057948 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.057926    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-token" (OuterVolumeSpecName: "istio-token") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:27:54.058035 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.057974    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" (UID: "f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:27:54.142655 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.142610    2552 generic.go:356] "Generic (PLEG): container finished" podID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerID="201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55" exitCode=0
Sep 10 12:27:54.142850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.142663    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"
Sep 10 12:27:54.142850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.142702    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" event={"ID":"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d","Type":"ContainerDied","Data":"201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55"}
Sep 10 12:27:54.142850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.142770    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg" event={"ID":"f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d","Type":"ContainerDied","Data":"7f31bf807f0dc66d381c68d504b7deadd1c35405387257e17b2f3e940cdc0d59"}
Sep 10 12:27:54.142850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.142799    2552 scope.go:117] "RemoveContainer" containerID="201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55"
Sep 10 12:27:54.151855 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.151840    2552 scope.go:117] "RemoveContainer" containerID="d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4"
Sep 10 12:27:54.156473 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.156455    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.156473 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.156473    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.156602 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.156481    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.156602 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.156489    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mpz27\" (UniqueName: \"kubernetes.io/projected/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d-kube-api-access-mpz27\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:27:54.160156 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.160141    2552 scope.go:117] "RemoveContainer" containerID="201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55"
Sep 10 12:27:54.160421 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:27:54.160403    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55\": container with ID starting with 201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55 not found: ID does not exist" containerID="201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55"
Sep 10 12:27:54.160481 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.160445    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55"} err="failed to get container status \"201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55\": rpc error: code = NotFound desc = could not find container \"201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55\": container with ID starting with 201767955c5ff4d36d8b700ddfc601f6ca83ff333b1cdcd03a269b6b4750ea55 not found: ID does not exist"
Sep 10 12:27:54.160481 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.160463    2552 scope.go:117] "RemoveContainer" containerID="d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4"
Sep 10 12:27:54.160703 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:27:54.160689    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4\": container with ID starting with d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4 not found: ID does not exist" containerID="d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4"
Sep 10 12:27:54.160741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.160708    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4"} err="failed to get container status \"d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4\": rpc error: code = NotFound desc = could not find container \"d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4\": container with ID starting with d28d837ded494fea5107622f055134b54b5cc63b3bb245aa9f8a7e9e533e32e4 not found: ID does not exist"
Sep 10 12:27:54.169058 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.169014    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"]
Sep 10 12:27:54.171044 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:54.171025    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-062bb857-openshift-default-664d988556-6t7qg"]
Sep 10 12:27:55.242451 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.242395    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" path="/var/lib/kubelet/pods/f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d/volumes"
Sep 10 12:27:55.274934 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.274907    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"]
Sep 10 12:27:55.275240 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.275226    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy"
Sep 10 12:27:55.275313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.275244    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy"
Sep 10 12:27:55.275313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.275265    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy"
Sep 10 12:27:55.275313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.275271    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy"
Sep 10 12:27:55.275412 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.275325    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy"
Sep 10 12:27:55.275412 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.275340    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3d26f41-c799-43d7-ad2c-3a1cd6e3d18d" containerName="istio-proxy"
Sep 10 12:27:55.279357 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.279343    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.282490 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.282474    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-f89a79\""
Sep 10 12:27:55.292359 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.292333    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"]
Sep 10 12:27:55.367819 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.367787    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkfl4\" (UniqueName: \"kubernetes.io/projected/7f1626bd-1dff-40a7-addd-d6d72ceb800c-kube-api-access-pkfl4\") pod \"payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.368001 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.367847    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1626bd-1dff-40a7-addd-d6d72ceb800c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.403837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.403810    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"]
Sep 10 12:27:55.407235 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.407213    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.409797 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.409774    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-f89a79-dockercfg-hwfzr\""
Sep 10 12:27:55.421276 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.421251    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"]
Sep 10 12:27:55.468324 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.468285    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1626bd-1dff-40a7-addd-d6d72ceb800c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.468485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.468380    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkfl4\" (UniqueName: \"kubernetes.io/projected/7f1626bd-1dff-40a7-addd-d6d72ceb800c-kube-api-access-pkfl4\") pod \"payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.468982 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.468959    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1626bd-1dff-40a7-addd-d6d72ceb800c-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.477979 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.477950    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkfl4\" (UniqueName: \"kubernetes.io/projected/7f1626bd-1dff-40a7-addd-d6d72ceb800c-kube-api-access-pkfl4\") pod \"payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.569643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.569546    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hsmdm\" (UniqueName: \"kubernetes.io/projected/a510b0fa-6547-41b0-9cfd-27696f31d1d1-kube-api-access-hsmdm\") pod \"payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.569643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.569588    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a510b0fa-6547-41b0-9cfd-27696f31d1d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.588726 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.588698    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:55.671043 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.671007    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hsmdm\" (UniqueName: \"kubernetes.io/projected/a510b0fa-6547-41b0-9cfd-27696f31d1d1-kube-api-access-hsmdm\") pod \"payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.671191 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.671052    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a510b0fa-6547-41b0-9cfd-27696f31d1d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.671816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.671796    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a510b0fa-6547-41b0-9cfd-27696f31d1d1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.682165 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.682145    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hsmdm\" (UniqueName: \"kubernetes.io/projected/a510b0fa-6547-41b0-9cfd-27696f31d1d1-kube-api-access-hsmdm\") pod \"payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.717161 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.717126    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:55.862801 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.862765    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"]
Sep 10 12:27:55.864945 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:27:55.864907    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f1626bd_1dff_40a7_addd_d6d72ceb800c.slice/crio-39e0203a2b6fa69285bff264a384022b2c543a6c2097f795b1e2604cddf8dc53 WatchSource:0}: Error finding container 39e0203a2b6fa69285bff264a384022b2c543a6c2097f795b1e2604cddf8dc53: Status 404 returned error can't find the container with id 39e0203a2b6fa69285bff264a384022b2c543a6c2097f795b1e2604cddf8dc53
Sep 10 12:27:55.992360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:55.992334    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"]
Sep 10 12:27:55.994778 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:27:55.994746    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda510b0fa_6547_41b0_9cfd_27696f31d1d1.slice/crio-0633e7279d373cb1220879f91df71d308e2ca16002ff698058d1b9ebed838e22 WatchSource:0}: Error finding container 0633e7279d373cb1220879f91df71d308e2ca16002ff698058d1b9ebed838e22: Status 404 returned error can't find the container with id 0633e7279d373cb1220879f91df71d308e2ca16002ff698058d1b9ebed838e22
Sep 10 12:27:56.155762 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.155728    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh" event={"ID":"a510b0fa-6547-41b0-9cfd-27696f31d1d1","Type":"ContainerStarted","Data":"7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46"}
Sep 10 12:27:56.155969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.155777    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh" event={"ID":"a510b0fa-6547-41b0-9cfd-27696f31d1d1","Type":"ContainerStarted","Data":"0633e7279d373cb1220879f91df71d308e2ca16002ff698058d1b9ebed838e22"}
Sep 10 12:27:56.155969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.155830    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:27:56.157561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.157538    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq" event={"ID":"7f1626bd-1dff-40a7-addd-d6d72ceb800c","Type":"ContainerStarted","Data":"bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991"}
Sep 10 12:27:56.157686 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.157574    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq" event={"ID":"7f1626bd-1dff-40a7-addd-d6d72ceb800c","Type":"ContainerStarted","Data":"39e0203a2b6fa69285bff264a384022b2c543a6c2097f795b1e2604cddf8dc53"}
Sep 10 12:27:56.157686 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.157592    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:27:56.175343 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.175299    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh" podStartSLOduration=1.175287004 podStartE2EDuration="1.175287004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:27:56.173820827 +0000 UTC m=+1087.370336538" watchObservedRunningTime="2026-09-10 12:27:56.175287004 +0000 UTC m=+1087.371802694"
Sep 10 12:27:56.193681 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:56.193644    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq" podStartSLOduration=1.193631519 podStartE2EDuration="1.193631519s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:27:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:27:56.192759587 +0000 UTC m=+1087.389275289" watchObservedRunningTime="2026-09-10 12:27:56.193631519 +0000 UTC m=+1087.390147219"
Sep 10 12:27:58.254465 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:27:58.254414    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:28:07.165251 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:07.165220    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:28:07.165733 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:07.165315    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:28:08.137407 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.137377    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 12:28:08.137593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.137458    2552 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 12:28:08.219718 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.219635    2552 generic.go:356] "Generic (PLEG): container finished" podID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerID="62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644" exitCode=0
Sep 10 12:28:08.219718 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.219705    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" event={"ID":"0b3e800d-e06e-4251-8dc7-77573950ade0","Type":"ContainerDied","Data":"62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644"}
Sep 10 12:28:08.220140 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.220069    2552 scope.go:117] "RemoveContainer" containerID="62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644"
Sep 10 12:28:08.221097 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.221072    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:08.221166 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.221135    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:08.221209 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:08.221165    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:09.232263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:09.232229    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" event={"ID":"0b3e800d-e06e-4251-8dc7-77573950ade0","Type":"ContainerStarted","Data":"4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874"}
Sep 10 12:28:09.682604 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:09.682572    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:28:09.682816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:09.682793    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 12:28:09.682864 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:09.682843    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 12:28:10.104193 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.104093    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"]
Sep 10 12:28:10.108475 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.108448    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.111666 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.111648    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-d1e553-dockercfg-z7hcw\""
Sep 10 12:28:10.112374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.112345    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-d1e553\""
Sep 10 12:28:10.124923 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.124902    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"]
Sep 10 12:28:10.195309 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.195280    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd6433c-735b-439f-b53b-8be16b6746c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.195539 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.195358    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ss6xr\" (UniqueName: \"kubernetes.io/projected/bbd6433c-735b-439f-b53b-8be16b6746c6-kube-api-access-ss6xr\") pod \"payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.296322 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.296279    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ss6xr\" (UniqueName: \"kubernetes.io/projected/bbd6433c-735b-439f-b53b-8be16b6746c6-kube-api-access-ss6xr\") pod \"payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.296838 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.296423    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd6433c-735b-439f-b53b-8be16b6746c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.297355 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.297326    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd6433c-735b-439f-b53b-8be16b6746c6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.307760 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.307712    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ss6xr\" (UniqueName: \"kubernetes.io/projected/bbd6433c-735b-439f-b53b-8be16b6746c6-kube-api-access-ss6xr\") pod \"payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.424236 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.424193    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:10.682271 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.682196    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 12:28:10.682271 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.682258    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 12:28:10.701796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:10.700248    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"]
Sep 10 12:28:10.716886 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:28:10.716848    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbbd6433c_735b_439f_b53b_8be16b6746c6.slice/crio-7bafd456c2eede029d87f41dab106b2ecde61b8bd78f33d7243d1342b4c3bf52 WatchSource:0}: Error finding container 7bafd456c2eede029d87f41dab106b2ecde61b8bd78f33d7243d1342b4c3bf52: Status 404 returned error can't find the container with id 7bafd456c2eede029d87f41dab106b2ecde61b8bd78f33d7243d1342b4c3bf52
Sep 10 12:28:11.244677 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:11.244646    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64" event={"ID":"bbd6433c-735b-439f-b53b-8be16b6746c6","Type":"ContainerStarted","Data":"e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610"}
Sep 10 12:28:11.244866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:11.244709    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:11.244866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:11.244732    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64" event={"ID":"bbd6433c-735b-439f-b53b-8be16b6746c6","Type":"ContainerStarted","Data":"7bafd456c2eede029d87f41dab106b2ecde61b8bd78f33d7243d1342b4c3bf52"}
Sep 10 12:28:11.264578 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:11.264530    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64" podStartSLOduration=1.264514038 podStartE2EDuration="1.264514038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:28:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:28:11.263290394 +0000 UTC m=+1102.459806094" watchObservedRunningTime="2026-09-10 12:28:11.264514038 +0000 UTC m=+1102.461029738"
Sep 10 12:28:11.682490 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:11.682417    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 12:28:11.682897 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:11.682512    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 12:28:12.450381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:12.450344    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:28:12.682348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:12.682311    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 12:28:12.682547 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:12.682376    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 12:28:13.682169 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:13.682131    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Sep 10 12:28:13.682324 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:13.682188    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Sep 10 12:28:14.686442 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:14.686401    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:28:14.686881 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:14.686700    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:28:14.687462 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:14.687445    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:28:21.046651 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.046606    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"]
Sep 10 12:28:21.050128 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.050104    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.054887 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.054860    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"auth-test-model-18c2d2-kserve-self-signed-certs\""
Sep 10 12:28:21.054887 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.054875    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"openshift-service-ca.crt\""
Sep 10 12:28:21.055045 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.054860    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"default-dockercfg-nlftf\""
Sep 10 12:28:21.055045 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.054927    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"kube-root-ca.crt\""
Sep 10 12:28:21.066651 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.066632    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"]
Sep 10 12:28:21.108745 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.108715    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-tmp-dir\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.108885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.108748    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-model-cache\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.108885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.108787    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88646d2a-1473-4e23-8950-d18e44ce7429-tls-certs\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.108885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.108809    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dznwj\" (UniqueName: \"kubernetes.io/projected/88646d2a-1473-4e23-8950-d18e44ce7429-kube-api-access-dznwj\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.108885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.108847    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-home\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.108885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.108861    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-dshm\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.209988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.209950    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-home\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.209988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.209988    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-dshm\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.210227 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.210059    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-tmp-dir\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.210227 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.210088    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-model-cache\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.210227 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.210146    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88646d2a-1473-4e23-8950-d18e44ce7429-tls-certs\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.210227 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.210172    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dznwj\" (UniqueName: \"kubernetes.io/projected/88646d2a-1473-4e23-8950-d18e44ce7429-kube-api-access-dznwj\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.210462 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.210414    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-home\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.210541 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.210524    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-tmp-dir\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.210589 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.210568    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-model-cache\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.212544 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.212523    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-dshm\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.212786 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.212769    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88646d2a-1473-4e23-8950-d18e44ce7429-tls-certs\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.219277 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.219255    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dznwj\" (UniqueName: \"kubernetes.io/projected/88646d2a-1473-4e23-8950-d18e44ce7429-kube-api-access-dznwj\") pod \"auth-test-model-18c2d2-kserve-7677946555-vzx67\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.359390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.359316    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:21.636786 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:21.636764    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"]
Sep 10 12:28:21.639440 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:28:21.639388    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88646d2a_1473_4e23_8950_d18e44ce7429.slice/crio-6e8c5336b2d2bde5a86f4d2b6cdb20d392f3497e78c2bdd285d5efcad7a9dedb WatchSource:0}: Error finding container 6e8c5336b2d2bde5a86f4d2b6cdb20d392f3497e78c2bdd285d5efcad7a9dedb: Status 404 returned error can't find the container with id 6e8c5336b2d2bde5a86f4d2b6cdb20d392f3497e78c2bdd285d5efcad7a9dedb
Sep 10 12:28:22.251853 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:22.251820    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:28:22.299112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:22.299078    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67" event={"ID":"88646d2a-1473-4e23-8950-d18e44ce7429","Type":"ContainerStarted","Data":"79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0"}
Sep 10 12:28:22.299335 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:22.299124    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67" event={"ID":"88646d2a-1473-4e23-8950-d18e44ce7429","Type":"ContainerStarted","Data":"6e8c5336b2d2bde5a86f4d2b6cdb20d392f3497e78c2bdd285d5efcad7a9dedb"}
Sep 10 12:28:22.318638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:22.318585    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67" podStartSLOduration=1.31856689 podStartE2EDuration="1.31856689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:28:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:28:22.317965924 +0000 UTC m=+1113.514481626" watchObservedRunningTime="2026-09-10 12:28:22.31856689 +0000 UTC m=+1113.515082595"
Sep 10 12:28:25.094698 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:25.094665    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:28:31.360488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:31.360457    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:31.360836 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:31.360517    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:31.373587 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:31.373561    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:32.357613 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:32.357582    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:28:33.487916 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.487880    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"]
Sep 10 12:28:33.492617 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.492601    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.495368 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.495346    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-a08ea2-openshift-default-dockercfg-9r7cr\""
Sep 10 12:28:33.509834 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.509809    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"]
Sep 10 12:28:33.632508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632465    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjzz9\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-kube-api-access-rjzz9\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632511    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-credential-socket\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632530    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-podinfo\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632560    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-certs\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632602    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-envoy\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632622    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-data\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632643    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632732    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-socket\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.632990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.632793    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-token\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.733990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.733954    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-token\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.733990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734002    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjzz9\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-kube-api-access-rjzz9\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734023    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-credential-socket\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734041    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-podinfo\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734062    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-certs\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734127    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-envoy\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734157    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-data\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734189    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734228    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-socket\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734687 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734510    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-credential-socket\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734687 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734544    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-certs\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734801 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734697    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-socket\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.734801 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.734763    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-data\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.735170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.735150    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istiod-ca-cert\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.736717 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.736688    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-envoy\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.736843 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.736815    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-podinfo\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.742918 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.742862    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjzz9\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-kube-api-access-rjzz9\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.746212 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.746192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-token\") pod \"e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") " pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:33.804490 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:33.804459    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:34.083880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.083854    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"]
Sep 10 12:28:34.085808 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:28:34.085783    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e963c5_8aea_42d7_bcb5_71be9e9331b0.slice/crio-d076e8ef02b505752381a25dd162f93be052a41207c96f307496ba094615de64 WatchSource:0}: Error finding container d076e8ef02b505752381a25dd162f93be052a41207c96f307496ba094615de64: Status 404 returned error can't find the container with id d076e8ef02b505752381a25dd162f93be052a41207c96f307496ba094615de64
Sep 10 12:28:34.088006 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.087971    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:34.088073 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.088058    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:34.088111 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.088099    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:34.357010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.356922    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" event={"ID":"18e963c5-8aea-42d7-bcb5-71be9e9331b0","Type":"ContainerStarted","Data":"7ad6408505c6c66cc7836f521b1b38bb5a4fcde4eaf1586b518b0dd1b552f819"}
Sep 10 12:28:34.357010 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.356992    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" event={"ID":"18e963c5-8aea-42d7-bcb5-71be9e9331b0","Type":"ContainerStarted","Data":"d076e8ef02b505752381a25dd162f93be052a41207c96f307496ba094615de64"}
Sep 10 12:28:34.397152 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.397090    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" podStartSLOduration=1.397069256 podStartE2EDuration="1.397069256s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:28:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:28:34.392627549 +0000 UTC m=+1125.589143249" watchObservedRunningTime="2026-09-10 12:28:34.397069256 +0000 UTC m=+1125.593584957"
Sep 10 12:28:34.805542 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:34.805507    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:35.810098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:35.810064    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:36.366667 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:36.366633    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:36.367650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:36.367627    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:38.569098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.569069    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"]
Sep 10 12:28:38.572277 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.572260    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:38.575002 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.574976    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-a08ea2-dockercfg-ghh57\""
Sep 10 12:28:38.575156 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.575136    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-a08ea2\""
Sep 10 12:28:38.585922 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.585896    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"]
Sep 10 12:28:38.680693 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.680662    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:38.680877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.680828    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szmjk\" (UniqueName: \"kubernetes.io/projected/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-kube-api-access-szmjk\") pod \"payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:38.781869 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.781837    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szmjk\" (UniqueName: \"kubernetes.io/projected/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-kube-api-access-szmjk\") pod \"payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:38.782036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.781885    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:38.782528 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.782510    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:38.791811 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.791785    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szmjk\" (UniqueName: \"kubernetes.io/projected/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-kube-api-access-szmjk\") pod \"payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:38.886185 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:38.886147    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:39.165548 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:39.165517    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"]
Sep 10 12:28:39.167720 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:28:39.167694    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3be77961_a2fb_4439_b06e_d27ecf3ff8b0.slice/crio-1dcb5f71421b09414d552d32183e5ba34ddc834875cb743b3d1f457397a05a84 WatchSource:0}: Error finding container 1dcb5f71421b09414d552d32183e5ba34ddc834875cb743b3d1f457397a05a84: Status 404 returned error can't find the container with id 1dcb5f71421b09414d552d32183e5ba34ddc834875cb743b3d1f457397a05a84
Sep 10 12:28:39.388398 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:39.388324    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5" event={"ID":"3be77961-a2fb-4439-b06e-d27ecf3ff8b0","Type":"ContainerStarted","Data":"e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4"}
Sep 10 12:28:39.388708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:39.388665    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5" event={"ID":"3be77961-a2fb-4439-b06e-d27ecf3ff8b0","Type":"ContainerStarted","Data":"1dcb5f71421b09414d552d32183e5ba34ddc834875cb743b3d1f457397a05a84"}
Sep 10 12:28:39.389842 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:39.389811    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:39.411969 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:39.411898    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5" podStartSLOduration=1.411884824 podStartE2EDuration="1.411884824s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:28:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:28:39.409937166 +0000 UTC m=+1130.606452867" watchObservedRunningTime="2026-09-10 12:28:39.411884824 +0000 UTC m=+1130.608400568"
Sep 10 12:28:45.250283 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:45.250240    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:28:49.341762 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:49.341732    2552 scope.go:117] "RemoveContainer" containerID="d217ace8c11b37dc9e7470074f754bea8d1b12a0582f15f5b904274966844897"
Sep 10 12:28:51.402643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:51.402601    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:28:56.477920 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:56.477884    2552 generic.go:356] "Generic (PLEG): container finished" podID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerID="7ad6408505c6c66cc7836f521b1b38bb5a4fcde4eaf1586b518b0dd1b552f819" exitCode=0
Sep 10 12:28:56.478454 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:56.477962    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" event={"ID":"18e963c5-8aea-42d7-bcb5-71be9e9331b0","Type":"ContainerDied","Data":"7ad6408505c6c66cc7836f521b1b38bb5a4fcde4eaf1586b518b0dd1b552f819"}
Sep 10 12:28:56.478542 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:56.478517    2552 scope.go:117] "RemoveContainer" containerID="7ad6408505c6c66cc7836f521b1b38bb5a4fcde4eaf1586b518b0dd1b552f819"
Sep 10 12:28:56.479678 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:56.479646    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:56.479784 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:56.479720    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:56.479784 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:56.479755    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:28:57.484659 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:57.484622    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" event={"ID":"18e963c5-8aea-42d7-bcb5-71be9e9331b0","Type":"ContainerStarted","Data":"e0261e62000ddbeb3da46d68533e283db7c51b296b89806ecd1fc7b34ff5d761"}
Sep 10 12:28:57.804883 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:57.804780    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:28:57.805090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:57.805059    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 12:28:57.805330 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:57.805300    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 12:28:58.805262 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:58.805213    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 12:28:58.805778 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:58.805295    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 12:28:59.805374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:59.805332    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 12:28:59.806038 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:28:59.806005    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 12:29:00.805106 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:00.805066    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 12:29:00.805406 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:00.805373    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 12:29:01.805949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:01.805909    2552 patch_prober.go:28] interesting pod/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body=
Sep 10 12:29:01.806572 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:01.806538    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused"
Sep 10 12:29:01.857051 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:01.857020    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:01.860571 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:01.860552    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:01.864719 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:01.864700    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-grlsn\""
Sep 10 12:29:01.870464 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:01.870441    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:02.019223 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.019175    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tvb4\" (UniqueName: \"kubernetes.io/projected/c29cb6f4-0c10-4333-b01c-555ca3e5dbfa-kube-api-access-9tvb4\") pod \"test-curl-7597-139740157502496\" (UID: \"c29cb6f4-0c10-4333-b01c-555ca3e5dbfa\") " pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:02.120322 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.120286    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tvb4\" (UniqueName: \"kubernetes.io/projected/c29cb6f4-0c10-4333-b01c-555ca3e5dbfa-kube-api-access-9tvb4\") pod \"test-curl-7597-139740157502496\" (UID: \"c29cb6f4-0c10-4333-b01c-555ca3e5dbfa\") " pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:02.144446 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.144400    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tvb4\" (UniqueName: \"kubernetes.io/projected/c29cb6f4-0c10-4333-b01c-555ca3e5dbfa-kube-api-access-9tvb4\") pod \"test-curl-7597-139740157502496\" (UID: \"c29cb6f4-0c10-4333-b01c-555ca3e5dbfa\") " pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:02.169469 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.169440    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:02.464475 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.464450    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:02.467740 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:29:02.467706    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc29cb6f4_0c10_4333_b01c_555ca3e5dbfa.slice/crio-a8ceed2c2e8a6b4d6489c339bdac8d37b7ba8c022bc72829ab358ab93e2d298f WatchSource:0}: Error finding container a8ceed2c2e8a6b4d6489c339bdac8d37b7ba8c022bc72829ab358ab93e2d298f: Status 404 returned error can't find the container with id a8ceed2c2e8a6b4d6489c339bdac8d37b7ba8c022bc72829ab358ab93e2d298f
Sep 10 12:29:02.468851 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.468831    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:29:02.515648 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.515608    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157502496" event={"ID":"c29cb6f4-0c10-4333-b01c-555ca3e5dbfa","Type":"ContainerStarted","Data":"a8ceed2c2e8a6b4d6489c339bdac8d37b7ba8c022bc72829ab358ab93e2d298f"}
Sep 10 12:29:02.809138 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.809057    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:29:02.809532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.809317    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:29:02.810111 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:02.810096    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:29:03.522269 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:03.522240    2552 generic.go:356] "Generic (PLEG): container finished" podID="c29cb6f4-0c10-4333-b01c-555ca3e5dbfa" containerID="dd066c969c338c8a0d6a7286eb910b879134fa1d69f0173819054ac330641aa6" exitCode=0
Sep 10 12:29:03.522394 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:03.522305    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157502496" event={"ID":"c29cb6f4-0c10-4333-b01c-555ca3e5dbfa","Type":"ContainerDied","Data":"dd066c969c338c8a0d6a7286eb910b879134fa1d69f0173819054ac330641aa6"}
Sep 10 12:29:04.652508 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.652485    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:04.690564 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.690532    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7597-139740157502496_c29cb6f4-0c10-4333-b01c-555ca3e5dbfa/test-curl-7597-139740157502496/0.log"
Sep 10 12:29:04.715958 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.715927    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:04.721528 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.721502    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:04.746708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.746684    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9tvb4\" (UniqueName: \"kubernetes.io/projected/c29cb6f4-0c10-4333-b01c-555ca3e5dbfa-kube-api-access-9tvb4\") pod \"c29cb6f4-0c10-4333-b01c-555ca3e5dbfa\" (UID: \"c29cb6f4-0c10-4333-b01c-555ca3e5dbfa\") "
Sep 10 12:29:04.748793 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.748767    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c29cb6f4-0c10-4333-b01c-555ca3e5dbfa-kube-api-access-9tvb4" (OuterVolumeSpecName: "kube-api-access-9tvb4") pod "c29cb6f4-0c10-4333-b01c-555ca3e5dbfa" (UID: "c29cb6f4-0c10-4333-b01c-555ca3e5dbfa"). InnerVolumeSpecName "kube-api-access-9tvb4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:29:04.840830 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.840760    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:04.841166 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.841152    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c29cb6f4-0c10-4333-b01c-555ca3e5dbfa" containerName="test-curl-7597-139740157502496"
Sep 10 12:29:04.841208 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.841171    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c29cb6f4-0c10-4333-b01c-555ca3e5dbfa" containerName="test-curl-7597-139740157502496"
Sep 10 12:29:04.841253 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.841241    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c29cb6f4-0c10-4333-b01c-555ca3e5dbfa" containerName="test-curl-7597-139740157502496"
Sep 10 12:29:04.844220 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.844205    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:04.847907 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.847881    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tvb4\" (UniqueName: \"kubernetes.io/projected/c29cb6f4-0c10-4333-b01c-555ca3e5dbfa-kube-api-access-9tvb4\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:29:04.850524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.850503    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:04.948635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:04.948602    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nc6z6\" (UniqueName: \"kubernetes.io/projected/41470694-7f3f-4d9c-a3cb-8f5476353e40-kube-api-access-nc6z6\") pod \"test-curl-7597-139740157502496\" (UID: \"41470694-7f3f-4d9c-a3cb-8f5476353e40\") " pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:05.050021 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.049983    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nc6z6\" (UniqueName: \"kubernetes.io/projected/41470694-7f3f-4d9c-a3cb-8f5476353e40-kube-api-access-nc6z6\") pod \"test-curl-7597-139740157502496\" (UID: \"41470694-7f3f-4d9c-a3cb-8f5476353e40\") " pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:05.059548 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.059518    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nc6z6\" (UniqueName: \"kubernetes.io/projected/41470694-7f3f-4d9c-a3cb-8f5476353e40-kube-api-access-nc6z6\") pod \"test-curl-7597-139740157502496\" (UID: \"41470694-7f3f-4d9c-a3cb-8f5476353e40\") " pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:05.154011 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.153982    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:05.242695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.242667    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c29cb6f4-0c10-4333-b01c-555ca3e5dbfa" path="/var/lib/kubelet/pods/c29cb6f4-0c10-4333-b01c-555ca3e5dbfa/volumes"
Sep 10 12:29:05.424025 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.424004    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:05.425950 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:29:05.425926    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41470694_7f3f_4d9c_a3cb_8f5476353e40.slice/crio-cd2ba745558dab5d06fac7b68713f60a63c12432f4ef4fbfaec1a2976ec2ef0a WatchSource:0}: Error finding container cd2ba745558dab5d06fac7b68713f60a63c12432f4ef4fbfaec1a2976ec2ef0a: Status 404 returned error can't find the container with id cd2ba745558dab5d06fac7b68713f60a63c12432f4ef4fbfaec1a2976ec2ef0a
Sep 10 12:29:05.533001 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.532976    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:05.533001 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.533001    2552 scope.go:117] "RemoveContainer" containerID="dd066c969c338c8a0d6a7286eb910b879134fa1d69f0173819054ac330641aa6"
Sep 10 12:29:05.534462 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:05.534415    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157502496" event={"ID":"41470694-7f3f-4d9c-a3cb-8f5476353e40","Type":"ContainerStarted","Data":"cd2ba745558dab5d06fac7b68713f60a63c12432f4ef4fbfaec1a2976ec2ef0a"}
Sep 10 12:29:06.541964 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:06.541884    2552 generic.go:356] "Generic (PLEG): container finished" podID="41470694-7f3f-4d9c-a3cb-8f5476353e40" containerID="ce8be83886395ac2d40ba27d7cce594dd235dbe9c559be46cba15470b9d2309e" exitCode=0
Sep 10 12:29:06.541964 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:06.541917    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157502496" event={"ID":"41470694-7f3f-4d9c-a3cb-8f5476353e40","Type":"ContainerDied","Data":"ce8be83886395ac2d40ba27d7cce594dd235dbe9c559be46cba15470b9d2309e"}
Sep 10 12:29:07.677395 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.677367    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:07.705794 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.705763    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7597-139740157502496_41470694-7f3f-4d9c-a3cb-8f5476353e40/test-curl-7597-139740157502496/0.log"
Sep 10 12:29:07.728511 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.728484    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:07.734588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.734567    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7597-139740157502496"]
Sep 10 12:29:07.776422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.776393    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nc6z6\" (UniqueName: \"kubernetes.io/projected/41470694-7f3f-4d9c-a3cb-8f5476353e40-kube-api-access-nc6z6\") pod \"41470694-7f3f-4d9c-a3cb-8f5476353e40\" (UID: \"41470694-7f3f-4d9c-a3cb-8f5476353e40\") "
Sep 10 12:29:07.778536 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.778508    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41470694-7f3f-4d9c-a3cb-8f5476353e40-kube-api-access-nc6z6" (OuterVolumeSpecName: "kube-api-access-nc6z6") pod "41470694-7f3f-4d9c-a3cb-8f5476353e40" (UID: "41470694-7f3f-4d9c-a3cb-8f5476353e40"). InnerVolumeSpecName "kube-api-access-nc6z6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:29:07.861444 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.861360    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:07.861805 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.861782    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41470694-7f3f-4d9c-a3cb-8f5476353e40" containerName="test-curl-7597-139740157502496"
Sep 10 12:29:07.861805 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.861805    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="41470694-7f3f-4d9c-a3cb-8f5476353e40" containerName="test-curl-7597-139740157502496"
Sep 10 12:29:07.861956 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.861888    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="41470694-7f3f-4d9c-a3cb-8f5476353e40" containerName="test-curl-7597-139740157502496"
Sep 10 12:29:07.864913 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.864895    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:07.874000 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.873976    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:07.877877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.877852    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nc6z6\" (UniqueName: \"kubernetes.io/projected/41470694-7f3f-4d9c-a3cb-8f5476353e40-kube-api-access-nc6z6\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:29:07.978825 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:07.978787    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/ee9830ec-48bb-4f66-90f1-c4adffcae95b-kube-api-access-fx6pc\") pod \"test-curl-7597-139740157503072\" (UID: \"ee9830ec-48bb-4f66-90f1-c4adffcae95b\") " pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:08.079677 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:08.079644    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/ee9830ec-48bb-4f66-90f1-c4adffcae95b-kube-api-access-fx6pc\") pod \"test-curl-7597-139740157503072\" (UID: \"ee9830ec-48bb-4f66-90f1-c4adffcae95b\") " pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:08.090222 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:08.090197    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/ee9830ec-48bb-4f66-90f1-c4adffcae95b-kube-api-access-fx6pc\") pod \"test-curl-7597-139740157503072\" (UID: \"ee9830ec-48bb-4f66-90f1-c4adffcae95b\") " pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:08.174292 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:08.174267    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:08.447997 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:08.447970    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:08.449717 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:29:08.449685    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podee9830ec_48bb_4f66_90f1_c4adffcae95b.slice/crio-071ce7b7f430aaf98fabad52fe2be8a5e4e6b4a41d07e18c27aaa32b74d95e04 WatchSource:0}: Error finding container 071ce7b7f430aaf98fabad52fe2be8a5e4e6b4a41d07e18c27aaa32b74d95e04: Status 404 returned error can't find the container with id 071ce7b7f430aaf98fabad52fe2be8a5e4e6b4a41d07e18c27aaa32b74d95e04
Sep 10 12:29:08.552611 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:08.552581    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:08.552611 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:08.552599    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cd2ba745558dab5d06fac7b68713f60a63c12432f4ef4fbfaec1a2976ec2ef0a"
Sep 10 12:29:08.553927 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:08.553890    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157503072" event={"ID":"ee9830ec-48bb-4f66-90f1-c4adffcae95b","Type":"ContainerStarted","Data":"071ce7b7f430aaf98fabad52fe2be8a5e4e6b4a41d07e18c27aaa32b74d95e04"}
Sep 10 12:29:08.563047 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:29:08.563021    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7597-139740157502496\" is forbidden: User \"system:node:ip-10-0-129-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-68.ec2.internal' and this object" podUID="41470694-7f3f-4d9c-a3cb-8f5476353e40" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:09.242158 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:29:09.242114    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7597-139740157502496\" is forbidden: User \"system:node:ip-10-0-129-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-68.ec2.internal' and this object" podUID="41470694-7f3f-4d9c-a3cb-8f5476353e40" pod="opendatahub/test-curl-7597-139740157502496"
Sep 10 12:29:09.243539 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:09.243513    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41470694-7f3f-4d9c-a3cb-8f5476353e40" path="/var/lib/kubelet/pods/41470694-7f3f-4d9c-a3cb-8f5476353e40/volumes"
Sep 10 12:29:09.559561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:09.559482    2552 generic.go:356] "Generic (PLEG): container finished" podID="ee9830ec-48bb-4f66-90f1-c4adffcae95b" containerID="fac56daa3a01ac212e5b4e719ff234007471f282e7287dbab027e456af30fdcb" exitCode=0
Sep 10 12:29:09.559714 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:09.559572    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157503072" event={"ID":"ee9830ec-48bb-4f66-90f1-c4adffcae95b","Type":"ContainerDied","Data":"fac56daa3a01ac212e5b4e719ff234007471f282e7287dbab027e456af30fdcb"}
Sep 10 12:29:10.690585 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.690563    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:10.714661 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.714608    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7597-139740157503072_ee9830ec-48bb-4f66-90f1-c4adffcae95b/test-curl-7597-139740157503072/0.log"
Sep 10 12:29:10.740525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.740496    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:10.744978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.744955    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:10.804077 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.804037    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/ee9830ec-48bb-4f66-90f1-c4adffcae95b-kube-api-access-fx6pc\") pod \"ee9830ec-48bb-4f66-90f1-c4adffcae95b\" (UID: \"ee9830ec-48bb-4f66-90f1-c4adffcae95b\") "
Sep 10 12:29:10.806202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.806176    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee9830ec-48bb-4f66-90f1-c4adffcae95b-kube-api-access-fx6pc" (OuterVolumeSpecName: "kube-api-access-fx6pc") pod "ee9830ec-48bb-4f66-90f1-c4adffcae95b" (UID: "ee9830ec-48bb-4f66-90f1-c4adffcae95b"). InnerVolumeSpecName "kube-api-access-fx6pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:29:10.866412 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.866389    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:10.866771 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.866758    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee9830ec-48bb-4f66-90f1-c4adffcae95b" containerName="test-curl-7597-139740157503072"
Sep 10 12:29:10.866821 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.866773    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee9830ec-48bb-4f66-90f1-c4adffcae95b" containerName="test-curl-7597-139740157503072"
Sep 10 12:29:10.866856 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.866836    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee9830ec-48bb-4f66-90f1-c4adffcae95b" containerName="test-curl-7597-139740157503072"
Sep 10 12:29:10.869825 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.869810    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:10.879090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.879065    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:10.905114 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:10.905089    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx6pc\" (UniqueName: \"kubernetes.io/projected/ee9830ec-48bb-4f66-90f1-c4adffcae95b-kube-api-access-fx6pc\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:29:11.006189 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.006152    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46rrk\" (UniqueName: \"kubernetes.io/projected/13689460-fd1a-44fa-ba3b-a6fc4244ef02-kube-api-access-46rrk\") pod \"test-curl-7597-139740157503072\" (UID: \"13689460-fd1a-44fa-ba3b-a6fc4244ef02\") " pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:11.107244 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.107207    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46rrk\" (UniqueName: \"kubernetes.io/projected/13689460-fd1a-44fa-ba3b-a6fc4244ef02-kube-api-access-46rrk\") pod \"test-curl-7597-139740157503072\" (UID: \"13689460-fd1a-44fa-ba3b-a6fc4244ef02\") " pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:11.116756 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.116689    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-46rrk\" (UniqueName: \"kubernetes.io/projected/13689460-fd1a-44fa-ba3b-a6fc4244ef02-kube-api-access-46rrk\") pod \"test-curl-7597-139740157503072\" (UID: \"13689460-fd1a-44fa-ba3b-a6fc4244ef02\") " pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:11.179525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.179499    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:11.244254 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.244221    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee9830ec-48bb-4f66-90f1-c4adffcae95b" path="/var/lib/kubelet/pods/ee9830ec-48bb-4f66-90f1-c4adffcae95b/volumes"
Sep 10 12:29:11.452458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.452417    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:11.454397 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:29:11.454372    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod13689460_fd1a_44fa_ba3b_a6fc4244ef02.slice/crio-9f3c67a1f5f4238bf6019eb2d6e7c6cff16a61eaf17511a3624830338aabbcb3 WatchSource:0}: Error finding container 9f3c67a1f5f4238bf6019eb2d6e7c6cff16a61eaf17511a3624830338aabbcb3: Status 404 returned error can't find the container with id 9f3c67a1f5f4238bf6019eb2d6e7c6cff16a61eaf17511a3624830338aabbcb3
Sep 10 12:29:11.570522 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.570492    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:11.570702 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.570500    2552 scope.go:117] "RemoveContainer" containerID="fac56daa3a01ac212e5b4e719ff234007471f282e7287dbab027e456af30fdcb"
Sep 10 12:29:11.571988 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:11.571731    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157503072" event={"ID":"13689460-fd1a-44fa-ba3b-a6fc4244ef02","Type":"ContainerStarted","Data":"9f3c67a1f5f4238bf6019eb2d6e7c6cff16a61eaf17511a3624830338aabbcb3"}
Sep 10 12:29:12.579525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:12.579498    2552 generic.go:356] "Generic (PLEG): container finished" podID="13689460-fd1a-44fa-ba3b-a6fc4244ef02" containerID="27863fa6940321223a0e22c9f0a2a0accf4837155459e57800e02d62a58b106b" exitCode=0
Sep 10 12:29:12.579857 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:12.579531    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7597-139740157503072" event={"ID":"13689460-fd1a-44fa-ba3b-a6fc4244ef02","Type":"ContainerDied","Data":"27863fa6940321223a0e22c9f0a2a0accf4837155459e57800e02d62a58b106b"}
Sep 10 12:29:13.722512 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.722487    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:13.753075 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.753044    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_test-curl-7597-139740157503072_13689460-fd1a-44fa-ba3b-a6fc4244ef02/test-curl-7597-139740157503072/0.log"
Sep 10 12:29:13.753727 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.753708    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:29:13.779518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.779493    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:13.788528 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.788500    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7597-139740157503072"]
Sep 10 12:29:13.830161 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.830139    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-46rrk\" (UniqueName: \"kubernetes.io/projected/13689460-fd1a-44fa-ba3b-a6fc4244ef02-kube-api-access-46rrk\") pod \"13689460-fd1a-44fa-ba3b-a6fc4244ef02\" (UID: \"13689460-fd1a-44fa-ba3b-a6fc4244ef02\") "
Sep 10 12:29:13.832290 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.832268    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13689460-fd1a-44fa-ba3b-a6fc4244ef02-kube-api-access-46rrk" (OuterVolumeSpecName: "kube-api-access-46rrk") pod "13689460-fd1a-44fa-ba3b-a6fc4244ef02" (UID: "13689460-fd1a-44fa-ba3b-a6fc4244ef02"). InnerVolumeSpecName "kube-api-access-46rrk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:29:13.931682 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:13.931643    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46rrk\" (UniqueName: \"kubernetes.io/projected/13689460-fd1a-44fa-ba3b-a6fc4244ef02-kube-api-access-46rrk\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:29:14.591808 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.591779    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:14.591978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.591819    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f3c67a1f5f4238bf6019eb2d6e7c6cff16a61eaf17511a3624830338aabbcb3"
Sep 10 12:29:14.605416 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:29:14.605381    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7597-139740157503072\" is forbidden: User \"system:node:ip-10-0-129-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-68.ec2.internal' and this object" podUID="13689460-fd1a-44fa-ba3b-a6fc4244ef02" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:14.746408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.746371    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"]
Sep 10 12:29:14.746885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.746816    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13689460-fd1a-44fa-ba3b-a6fc4244ef02" containerName="test-curl-7597-139740157503072"
Sep 10 12:29:14.746885 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.746830    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="13689460-fd1a-44fa-ba3b-a6fc4244ef02" containerName="test-curl-7597-139740157503072"
Sep 10 12:29:14.747002 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.746899    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="13689460-fd1a-44fa-ba3b-a6fc4244ef02" containerName="test-curl-7597-139740157503072"
Sep 10 12:29:14.749778 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.749761    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.752395 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:29:14.752363    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7597-139740157503072\" is forbidden: User \"system:node:ip-10-0-129-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-68.ec2.internal' and this object" podUID="13689460-fd1a-44fa-ba3b-a6fc4244ef02" pod="opendatahub/test-curl-7597-139740157503072"
Sep 10 12:29:14.752976 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.752958    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-156d6252-openshift-default-dockercfg-wp8wx\""
Sep 10 12:29:14.774772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.774749    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"]
Sep 10 12:29:14.841189 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841156    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-certs\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841218    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-socket\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841236    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istiod-ca-cert\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841273    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-credential-socket\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841329    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-podinfo\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841358 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841352    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-data\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841564 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841422    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h6btk\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-kube-api-access-h6btk\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841564 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841499    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-envoy\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.841564 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.841555    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-token\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942257 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942225    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-podinfo\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942266    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-data\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942292    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h6btk\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-kube-api-access-h6btk\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942323    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-envoy\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942352    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-token\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942368    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-certs\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942416    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-socket\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942547    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istiod-ca-cert\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942584    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-credential-socket\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942881 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942780    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-data\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942934 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942895    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-certs\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.942934 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.942916    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-credential-socket\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.943062 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.943040    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-socket\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.943335 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.943316    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istiod-ca-cert\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.944753 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.944732    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-envoy\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.944871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.944855    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-podinfo\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.962257 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.962226    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h6btk\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-kube-api-access-h6btk\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:14.967769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:14.967751    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-token\") pod \"e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") " pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:15.059896 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.059867    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:15.243619 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.243541    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13689460-fd1a-44fa-ba3b-a6fc4244ef02" path="/var/lib/kubelet/pods/13689460-fd1a-44fa-ba3b-a6fc4244ef02/volumes"
Sep 10 12:29:15.339279 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.339255    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"]
Sep 10 12:29:15.341149 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:29:15.341123    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd77ed0b6_db56_4fb3_8585_42c3d3e2cc79.slice/crio-2b45983eb907e47bdb9ac93568c0a3c60ba2f2bef8439c85fa0a54d7003a05ec WatchSource:0}: Error finding container 2b45983eb907e47bdb9ac93568c0a3c60ba2f2bef8439c85fa0a54d7003a05ec: Status 404 returned error can't find the container with id 2b45983eb907e47bdb9ac93568c0a3c60ba2f2bef8439c85fa0a54d7003a05ec
Sep 10 12:29:15.343354 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.343322    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:29:15.343470 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.343385    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:29:15.343470 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.343413    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:29:15.599328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.599241    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" event={"ID":"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79","Type":"ContainerStarted","Data":"d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa"}
Sep 10 12:29:15.599328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.599306    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" event={"ID":"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79","Type":"ContainerStarted","Data":"2b45983eb907e47bdb9ac93568c0a3c60ba2f2bef8439c85fa0a54d7003a05ec"}
Sep 10 12:29:15.624049 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:15.623997    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" podStartSLOduration=1.6239777050000002 podStartE2EDuration="1.623977705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:29:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:29:15.623294107 +0000 UTC m=+1166.819809808" watchObservedRunningTime="2026-09-10 12:29:15.623977705 +0000 UTC m=+1166.820493409"
Sep 10 12:29:16.060824 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:16.060785    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:16.066291 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:16.066267    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:16.605520 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:16.605485    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:16.606524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:16.606503    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:19.559542 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.559506    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"]
Sep 10 12:29:19.562997 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.562973    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:19.566692 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.566673    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-156d6252\""
Sep 10 12:29:19.567537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.567520    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-156d6252-dockercfg-l89m9\""
Sep 10 12:29:19.584333 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.584308    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"]
Sep 10 12:29:19.689282 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.689247    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:19.689484 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.689299    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx2d9\" (UniqueName: \"kubernetes.io/projected/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-kube-api-access-wx2d9\") pod \"payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:19.790404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.790366    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx2d9\" (UniqueName: \"kubernetes.io/projected/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-kube-api-access-wx2d9\") pod \"payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:19.790571 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.790503    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:19.791125 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.791107    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:19.800193 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.800153    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx2d9\" (UniqueName: \"kubernetes.io/projected/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-kube-api-access-wx2d9\") pod \"payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:19.873588 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:19.873560    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:20.149955 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:20.149883    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"]
Sep 10 12:29:20.153263 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:29:20.153238    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod903d25fa_3bf4_4c6a_ae26_2ac70b2cda48.slice/crio-7aea9ac8df7aeb21096b080f21eb07c234b2c54fad3497e7839aab178935e4c5 WatchSource:0}: Error finding container 7aea9ac8df7aeb21096b080f21eb07c234b2c54fad3497e7839aab178935e4c5: Status 404 returned error can't find the container with id 7aea9ac8df7aeb21096b080f21eb07c234b2c54fad3497e7839aab178935e4c5
Sep 10 12:29:20.636793 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:20.636757    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6" event={"ID":"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48","Type":"ContainerStarted","Data":"82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596"}
Sep 10 12:29:20.637181 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:20.636817    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6" event={"ID":"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48","Type":"ContainerStarted","Data":"7aea9ac8df7aeb21096b080f21eb07c234b2c54fad3497e7839aab178935e4c5"}
Sep 10 12:29:20.637181 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:20.636846    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:20.660877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:20.660819    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6" podStartSLOduration=1.6607976629999999 podStartE2EDuration="1.660797663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:29:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:29:20.660715435 +0000 UTC m=+1171.857231136" watchObservedRunningTime="2026-09-10 12:29:20.660797663 +0000 UTC m=+1171.857313366"
Sep 10 12:29:31.644118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:31.644085    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:29:33.353317 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:33.353279    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:29:36.720943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:36.720911    2552 generic.go:356] "Generic (PLEG): container finished" podID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerID="d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa" exitCode=0
Sep 10 12:29:36.721359 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:36.720954    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" event={"ID":"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79","Type":"ContainerDied","Data":"d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa"}
Sep 10 12:29:36.721359 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:36.721288    2552 scope.go:117] "RemoveContainer" containerID="d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa"
Sep 10 12:29:36.722365 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:36.722333    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:29:36.722416 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:36.722406    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:29:36.722468 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:36.722453    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:29:37.728087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:37.728051    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" event={"ID":"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79","Type":"ContainerStarted","Data":"fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505"}
Sep 10 12:29:38.060471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:38.060365    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:38.060637 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:38.060614    2552 patch_prober.go:28] interesting pod/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 12:29:38.060695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:38.060675    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 12:29:39.061023 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:39.060990    2552 patch_prober.go:28] interesting pod/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 12:29:39.061539 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:39.061071    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 12:29:40.060598 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:40.060562    2552 patch_prober.go:28] interesting pod/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 12:29:40.060786 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:40.060615    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 12:29:41.061239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:41.061201    2552 patch_prober.go:28] interesting pod/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 12:29:41.061641 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:41.061271    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 12:29:42.060715 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:42.060679    2552 patch_prober.go:28] interesting pod/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Sep 10 12:29:42.060890 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:42.060731    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Sep 10 12:29:43.065455 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:43.065405    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:43.065816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:43.065746    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:43.066349 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:43.066334    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:29:49.365185 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:49.365153    2552 scope.go:117] "RemoveContainer" containerID="b992249d96ade00e6a0bd3811be05846d73059cd47b8001a77a266c74ffdad93"
Sep 10 12:29:49.749554 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:29:49.749527    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:30:00.174814 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.174779    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"]
Sep 10 12:30:00.182197 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.182174    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"]
Sep 10 12:30:00.182356 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.182294    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"
Sep 10 12:30:00.185281 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.185263    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"
Sep 10 12:30:00.188321 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.188212    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:30:00.192943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.192925    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"
Sep 10 12:30:00.199566 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.199549    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"
Sep 10 12:30:00.206222 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.206202    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"]
Sep 10 12:30:00.213872 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.213850    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"]
Sep 10 12:30:00.219816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.219794    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"]
Sep 10 12:30:00.223724 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.223695    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"
Sep 10 12:30:00.237501 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.237169    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"]
Sep 10 12:30:00.251099 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.251074    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"
Sep 10 12:30:00.489239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.489214    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"]
Sep 10 12:30:00.491016 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:30:00.490995    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8507d204_52b0_487b_9d06_2af3dc3aeaaa.slice/crio-50cbf4c0590daab908d92f2d1a5854e9de62b325a455486a78afd3ec591126f5 WatchSource:0}: Error finding container 50cbf4c0590daab908d92f2d1a5854e9de62b325a455486a78afd3ec591126f5: Status 404 returned error can't find the container with id 50cbf4c0590daab908d92f2d1a5854e9de62b325a455486a78afd3ec591126f5
Sep 10 12:30:00.508374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.508350    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"]
Sep 10 12:30:00.510751 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:30:00.510719    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod737daabd_3c17_4b83_9de6_0a2a6dd5d6a6.slice/crio-bbe23d94644192b7f43f79cde3e3c9545bf13c43251985f735fce0f0c87b0c30 WatchSource:0}: Error finding container bbe23d94644192b7f43f79cde3e3c9545bf13c43251985f735fce0f0c87b0c30: Status 404 returned error can't find the container with id bbe23d94644192b7f43f79cde3e3c9545bf13c43251985f735fce0f0c87b0c30
Sep 10 12:30:00.553632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.553610    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"]
Sep 10 12:30:00.555618 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:30:00.555590    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00bf1aa6_faf2_49a8_ab42_75e107726345.slice/crio-1c4dbfa01eb0655881a1437421559a7134a3377c5dcb4c32f94b262d4c19868a WatchSource:0}: Error finding container 1c4dbfa01eb0655881a1437421559a7134a3377c5dcb4c32f94b262d4c19868a: Status 404 returned error can't find the container with id 1c4dbfa01eb0655881a1437421559a7134a3377c5dcb4c32f94b262d4c19868a
Sep 10 12:30:00.846747 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.846692    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l" event={"ID":"00bf1aa6-faf2-49a8-ab42-75e107726345","Type":"ContainerStarted","Data":"0bdba2d15c547b8e547748e764edbf74669a705879a9a09d0b4dd064e7fa52ac"}
Sep 10 12:30:00.846861 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.846746    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l" event={"ID":"00bf1aa6-faf2-49a8-ab42-75e107726345","Type":"ContainerStarted","Data":"1c4dbfa01eb0655881a1437421559a7134a3377c5dcb4c32f94b262d4c19868a"}
Sep 10 12:30:00.847983 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.847960    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956" event={"ID":"737daabd-3c17-4b83-9de6-0a2a6dd5d6a6","Type":"ContainerStarted","Data":"bbe23d94644192b7f43f79cde3e3c9545bf13c43251985f735fce0f0c87b0c30"}
Sep 10 12:30:00.849411 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.849389    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk" event={"ID":"8507d204-52b0-487b-9d06-2af3dc3aeaaa","Type":"ContainerStarted","Data":"64ce06c54b8c6103aa3e3e6635717fac1210bff4294293da7c136e07593a1bbb"}
Sep 10 12:30:00.849516 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.849444    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk" event={"ID":"8507d204-52b0-487b-9d06-2af3dc3aeaaa","Type":"ContainerStarted","Data":"50cbf4c0590daab908d92f2d1a5854e9de62b325a455486a78afd3ec591126f5"}
Sep 10 12:30:00.865256 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.865219    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l" podStartSLOduration=0.865204926 podStartE2EDuration="865.204926ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:00.863870897 +0000 UTC m=+1212.060386599" watchObservedRunningTime="2026-09-10 12:30:00.865204926 +0000 UTC m=+1212.061720625"
Sep 10 12:30:00.885712 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:00.885675    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk" podStartSLOduration=0.885663813 podStartE2EDuration="885.663813ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:00.882641556 +0000 UTC m=+1212.079157238" watchObservedRunningTime="2026-09-10 12:30:00.885663813 +0000 UTC m=+1212.082179513"
Sep 10 12:30:01.856145 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:01.856072    2552 generic.go:356] "Generic (PLEG): container finished" podID="00bf1aa6-faf2-49a8-ab42-75e107726345" containerID="0bdba2d15c547b8e547748e764edbf74669a705879a9a09d0b4dd064e7fa52ac" exitCode=0
Sep 10 12:30:01.856500 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:01.856153    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l" event={"ID":"00bf1aa6-faf2-49a8-ab42-75e107726345","Type":"ContainerDied","Data":"0bdba2d15c547b8e547748e764edbf74669a705879a9a09d0b4dd064e7fa52ac"}
Sep 10 12:30:01.857757 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:01.857735    2552 generic.go:356] "Generic (PLEG): container finished" podID="737daabd-3c17-4b83-9de6-0a2a6dd5d6a6" containerID="a9be319146f76cdb727ccc6cf9a3e598d14ce579419fb4c31253d879f4ec49e5" exitCode=0
Sep 10 12:30:01.857871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:01.857797    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956" event={"ID":"737daabd-3c17-4b83-9de6-0a2a6dd5d6a6","Type":"ContainerDied","Data":"a9be319146f76cdb727ccc6cf9a3e598d14ce579419fb4c31253d879f4ec49e5"}
Sep 10 12:30:01.859172 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:01.859150    2552 generic.go:356] "Generic (PLEG): container finished" podID="8507d204-52b0-487b-9d06-2af3dc3aeaaa" containerID="64ce06c54b8c6103aa3e3e6635717fac1210bff4294293da7c136e07593a1bbb" exitCode=0
Sep 10 12:30:01.859257 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:01.859186    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk" event={"ID":"8507d204-52b0-487b-9d06-2af3dc3aeaaa","Type":"ContainerDied","Data":"64ce06c54b8c6103aa3e3e6635717fac1210bff4294293da7c136e07593a1bbb"}
Sep 10 12:30:03.050987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.050964    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"
Sep 10 12:30:03.054317 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.054292    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"
Sep 10 12:30:03.078644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.078614    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"
Sep 10 12:30:03.160882 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.160855    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"]
Sep 10 12:30:03.161278 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161262    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8507d204-52b0-487b-9d06-2af3dc3aeaaa" containerName="cleanup"
Sep 10 12:30:03.161386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161281    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8507d204-52b0-487b-9d06-2af3dc3aeaaa" containerName="cleanup"
Sep 10 12:30:03.161386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161306    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="737daabd-3c17-4b83-9de6-0a2a6dd5d6a6" containerName="cleanup"
Sep 10 12:30:03.161386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161316    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="737daabd-3c17-4b83-9de6-0a2a6dd5d6a6" containerName="cleanup"
Sep 10 12:30:03.161386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161333    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00bf1aa6-faf2-49a8-ab42-75e107726345" containerName="cleanup"
Sep 10 12:30:03.161386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161341    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="00bf1aa6-faf2-49a8-ab42-75e107726345" containerName="cleanup"
Sep 10 12:30:03.161666 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161461    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="737daabd-3c17-4b83-9de6-0a2a6dd5d6a6" containerName="cleanup"
Sep 10 12:30:03.161666 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161478    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8507d204-52b0-487b-9d06-2af3dc3aeaaa" containerName="cleanup"
Sep 10 12:30:03.161666 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.161487    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="00bf1aa6-faf2-49a8-ab42-75e107726345" containerName="cleanup"
Sep 10 12:30:03.164450 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.164417    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.167241 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.167213    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-156d6252\"/\"kube-root-ca.crt\""
Sep 10 12:30:03.167345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.167293    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-156d6252\"/\"openshift-service-ca.crt\""
Sep 10 12:30:03.167528 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.167505    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-156d6252\"/\"e2e-lifecycle-model-156d6252-kserve-self-signed-certs\""
Sep 10 12:30:03.167619 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.167530    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-156d6252\"/\"default-dockercfg-bvmq6\""
Sep 10 12:30:03.175240 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.175218    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"]
Sep 10 12:30:03.285944 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.285914    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76f088d-58ab-4677-99f3-073e23c2fd13-tls-certs\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.286078 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.285989    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-model-cache\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.286078 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.286012    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-home\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.286078 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.286048    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-dshm\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.286301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.286086    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-tmp-dir\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.286301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.286196    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ptl7z\" (UniqueName: \"kubernetes.io/projected/a76f088d-58ab-4677-99f3-073e23c2fd13-kube-api-access-ptl7z\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387082    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76f088d-58ab-4677-99f3-073e23c2fd13-tls-certs\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387132    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-model-cache\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387154    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-home\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387175    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-dshm\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387207    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-tmp-dir\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387313 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387269    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ptl7z\" (UniqueName: \"kubernetes.io/projected/a76f088d-58ab-4677-99f3-073e23c2fd13-kube-api-access-ptl7z\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387624 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387592    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-home\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387624 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387608    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-tmp-dir\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.387704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.387687    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-model-cache\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.389551 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.389531    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-dshm\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.389826 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.389798    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76f088d-58ab-4677-99f3-073e23c2fd13-tls-certs\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.396632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.396606    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ptl7z\" (UniqueName: \"kubernetes.io/projected/a76f088d-58ab-4677-99f3-073e23c2fd13-kube-api-access-ptl7z\") pod \"e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") " pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.475309 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.475250    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:03.749590 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.749565    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"]
Sep 10 12:30:03.751488 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:30:03.751461    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda76f088d_58ab_4677_99f3_073e23c2fd13.slice/crio-c0987e76f3f1d829c7304c3287f654378300262b90e1e3fc43f2145be4fd8993 WatchSource:0}: Error finding container c0987e76f3f1d829c7304c3287f654378300262b90e1e3fc43f2145be4fd8993: Status 404 returned error can't find the container with id c0987e76f3f1d829c7304c3287f654378300262b90e1e3fc43f2145be4fd8993
Sep 10 12:30:03.871694 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.871668    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"
Sep 10 12:30:03.871817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.871712    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk" event={"ID":"8507d204-52b0-487b-9d06-2af3dc3aeaaa","Type":"ContainerDied","Data":"50cbf4c0590daab908d92f2d1a5854e9de62b325a455486a78afd3ec591126f5"}
Sep 10 12:30:03.871817 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.871771    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="50cbf4c0590daab908d92f2d1a5854e9de62b325a455486a78afd3ec591126f5"
Sep 10 12:30:03.873107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.873082    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l" event={"ID":"00bf1aa6-faf2-49a8-ab42-75e107726345","Type":"ContainerDied","Data":"1c4dbfa01eb0655881a1437421559a7134a3377c5dcb4c32f94b262d4c19868a"}
Sep 10 12:30:03.873107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.873102    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"
Sep 10 12:30:03.873297 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.873124    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c4dbfa01eb0655881a1437421559a7134a3377c5dcb4c32f94b262d4c19868a"
Sep 10 12:30:03.874928 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.874902    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5" event={"ID":"a76f088d-58ab-4677-99f3-073e23c2fd13","Type":"ContainerStarted","Data":"6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0"}
Sep 10 12:30:03.875050 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.874940    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5" event={"ID":"a76f088d-58ab-4677-99f3-073e23c2fd13","Type":"ContainerStarted","Data":"c0987e76f3f1d829c7304c3287f654378300262b90e1e3fc43f2145be4fd8993"}
Sep 10 12:30:03.876385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.876367    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956" event={"ID":"737daabd-3c17-4b83-9de6-0a2a6dd5d6a6","Type":"ContainerDied","Data":"bbe23d94644192b7f43f79cde3e3c9545bf13c43251985f735fce0f0c87b0c30"}
Sep 10 12:30:03.876498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.876395    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"
Sep 10 12:30:03.876498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.876398    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbe23d94644192b7f43f79cde3e3c9545bf13c43251985f735fce0f0c87b0c30"
Sep 10 12:30:03.896831 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:03.896782    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5" podStartSLOduration=0.896763992 podStartE2EDuration="896.763992ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:30:03.895141761 +0000 UTC m=+1215.091657463" watchObservedRunningTime="2026-09-10 12:30:03.896763992 +0000 UTC m=+1215.093279693"
Sep 10 12:30:06.850198 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:06.850160    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:30:13.475754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:13.475719    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:13.476154 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:13.475794    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:13.488610 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:13.488591    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:13.939496 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:13.939473    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:30:23.750477 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:23.750444    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:30:41.848776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:41.848689    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:30:42.313266 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.313222    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"]
Sep 10 12:30:42.316830 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.316806    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-mt-156d6252-29817390-vl956"]
Sep 10 12:30:42.370882 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.370855    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"]
Sep 10 12:30:42.371109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.371074    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6" podUID="903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" containerName="payload-pre-processing" containerID="cri-o://82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596" gracePeriod=30
Sep 10 12:30:42.623528 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.623502    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:30:42.727519 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.727480    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wx2d9\" (UniqueName: \"kubernetes.io/projected/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-kube-api-access-wx2d9\") pod \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") "
Sep 10 12:30:42.727727 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.727554    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-plugins-config-volume\") pod \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\" (UID: \"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48\") "
Sep 10 12:30:42.727906 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.727880    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" (UID: "903d25fa-3bf4-4c6a-ae26-2ac70b2cda48"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:30:42.729741 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.729714    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-kube-api-access-wx2d9" (OuterVolumeSpecName: "kube-api-access-wx2d9") pod "903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" (UID: "903d25fa-3bf4-4c6a-ae26-2ac70b2cda48"). InnerVolumeSpecName "kube-api-access-wx2d9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:30:42.829239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.829200    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:30:42.829239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:42.829241    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx2d9\" (UniqueName: \"kubernetes.io/projected/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48-kube-api-access-wx2d9\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:30:43.074441 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.074406    2552 generic.go:356] "Generic (PLEG): container finished" podID="903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" containerID="82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596" exitCode=0
Sep 10 12:30:43.074866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.074461    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6" event={"ID":"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48","Type":"ContainerDied","Data":"82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596"}
Sep 10 12:30:43.074866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.074481    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"
Sep 10 12:30:43.074866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.074529    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6" event={"ID":"903d25fa-3bf4-4c6a-ae26-2ac70b2cda48","Type":"ContainerDied","Data":"7aea9ac8df7aeb21096b080f21eb07c234b2c54fad3497e7839aab178935e4c5"}
Sep 10 12:30:43.074866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.074554    2552 scope.go:117] "RemoveContainer" containerID="82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596"
Sep 10 12:30:43.083593 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.083578    2552 scope.go:117] "RemoveContainer" containerID="82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596"
Sep 10 12:30:43.083873 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:30:43.083853    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596\": container with ID starting with 82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596 not found: ID does not exist" containerID="82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596"
Sep 10 12:30:43.083927 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.083880    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596"} err="failed to get container status \"82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596\": rpc error: code = NotFound desc = could not find container \"82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596\": container with ID starting with 82ef4a7d38a8c0878f0eee4a771550faf2432122d32dbecb13f2ccad8778a596 not found: ID does not exist"
Sep 10 12:30:43.099170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.099146    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"]
Sep 10 12:30:43.103296 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.103277    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-156d6252-6f67784cf8-8tpm6"]
Sep 10 12:30:43.242683 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.242658    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="737daabd-3c17-4b83-9de6-0a2a6dd5d6a6" path="/var/lib/kubelet/pods/737daabd-3c17-4b83-9de6-0a2a6dd5d6a6/volumes"
Sep 10 12:30:43.242982 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:43.242969    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" path="/var/lib/kubelet/pods/903d25fa-3bf4-4c6a-ae26-2ac70b2cda48/volumes"
Sep 10 12:30:49.395481 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:49.395447    2552 scope.go:117] "RemoveContainer" containerID="0692518e07c18434fd82cbfd6ca2581446047cddcfd08fca33c85a0c9d824093"
Sep 10 12:30:50.612241 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:50.612207    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"]
Sep 10 12:30:57.447103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:30:57.447069    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:31:05.632128 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.632091    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5" podUID="a76f088d-58ab-4677-99f3-073e23c2fd13" containerName="main" containerID="cri-o://6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0" gracePeriod=45
Sep 10 12:31:05.888923 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.888870    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:31:05.930350 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930320    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-dshm\") pod \"a76f088d-58ab-4677-99f3-073e23c2fd13\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") "
Sep 10 12:31:05.930532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930382    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76f088d-58ab-4677-99f3-073e23c2fd13-tls-certs\") pod \"a76f088d-58ab-4677-99f3-073e23c2fd13\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") "
Sep 10 12:31:05.930532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930508    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-home\") pod \"a76f088d-58ab-4677-99f3-073e23c2fd13\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") "
Sep 10 12:31:05.930653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930537    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-model-cache\") pod \"a76f088d-58ab-4677-99f3-073e23c2fd13\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") "
Sep 10 12:31:05.930653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930565    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-tmp-dir\") pod \"a76f088d-58ab-4677-99f3-073e23c2fd13\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") "
Sep 10 12:31:05.930653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930599    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ptl7z\" (UniqueName: \"kubernetes.io/projected/a76f088d-58ab-4677-99f3-073e23c2fd13-kube-api-access-ptl7z\") pod \"a76f088d-58ab-4677-99f3-073e23c2fd13\" (UID: \"a76f088d-58ab-4677-99f3-073e23c2fd13\") "
Sep 10 12:31:05.931096 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930798    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-home" (OuterVolumeSpecName: "home") pod "a76f088d-58ab-4677-99f3-073e23c2fd13" (UID: "a76f088d-58ab-4677-99f3-073e23c2fd13"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:05.931096 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930853    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-model-cache" (OuterVolumeSpecName: "model-cache") pod "a76f088d-58ab-4677-99f3-073e23c2fd13" (UID: "a76f088d-58ab-4677-99f3-073e23c2fd13"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:05.931096 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.930972    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:05.931096 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.931009    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:05.931096 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.931050    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a76f088d-58ab-4677-99f3-073e23c2fd13" (UID: "a76f088d-58ab-4677-99f3-073e23c2fd13"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:05.932908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.932883    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-dshm" (OuterVolumeSpecName: "dshm") pod "a76f088d-58ab-4677-99f3-073e23c2fd13" (UID: "a76f088d-58ab-4677-99f3-073e23c2fd13"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:05.933116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.933088    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a76f088d-58ab-4677-99f3-073e23c2fd13-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a76f088d-58ab-4677-99f3-073e23c2fd13" (UID: "a76f088d-58ab-4677-99f3-073e23c2fd13"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:31:05.933247 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:05.933228    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a76f088d-58ab-4677-99f3-073e23c2fd13-kube-api-access-ptl7z" (OuterVolumeSpecName: "kube-api-access-ptl7z") pod "a76f088d-58ab-4677-99f3-073e23c2fd13" (UID: "a76f088d-58ab-4677-99f3-073e23c2fd13"). InnerVolumeSpecName "kube-api-access-ptl7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:31:06.032124 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.032096    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:06.032124 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.032121    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a76f088d-58ab-4677-99f3-073e23c2fd13-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:06.032124 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.032131    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a76f088d-58ab-4677-99f3-073e23c2fd13-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:06.032355 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.032139    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ptl7z\" (UniqueName: \"kubernetes.io/projected/a76f088d-58ab-4677-99f3-073e23c2fd13-kube-api-access-ptl7z\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:06.194893 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.194817    2552 generic.go:356] "Generic (PLEG): container finished" podID="a76f088d-58ab-4677-99f3-073e23c2fd13" containerID="6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0" exitCode=0
Sep 10 12:31:06.194893 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.194857    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"
Sep 10 12:31:06.195062 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.194898    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5" event={"ID":"a76f088d-58ab-4677-99f3-073e23c2fd13","Type":"ContainerDied","Data":"6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0"}
Sep 10 12:31:06.195062 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.194941    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5" event={"ID":"a76f088d-58ab-4677-99f3-073e23c2fd13","Type":"ContainerDied","Data":"c0987e76f3f1d829c7304c3287f654378300262b90e1e3fc43f2145be4fd8993"}
Sep 10 12:31:06.195062 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.194959    2552 scope.go:117] "RemoveContainer" containerID="6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0"
Sep 10 12:31:06.204199 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.204182    2552 scope.go:117] "RemoveContainer" containerID="6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0"
Sep 10 12:31:06.204487 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:31:06.204467    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0\": container with ID starting with 6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0 not found: ID does not exist" containerID="6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0"
Sep 10 12:31:06.204590 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.204493    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0"} err="failed to get container status \"6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0\": rpc error: code = NotFound desc = could not find container \"6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0\": container with ID starting with 6bcc3350e8ad91cb055c933f77dc63a2c7a2368ffa75cc91b914b1f99288e7c0 not found: ID does not exist"
Sep 10 12:31:06.219626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.219605    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"]
Sep 10 12:31:06.231247 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:06.231230    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-156d6252/e2e-lifecycle-model-156d6252-kserve-5d74dd99dc-mp4l5"]
Sep 10 12:31:07.241729 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:07.241694    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a76f088d-58ab-4677-99f3-073e23c2fd13" path="/var/lib/kubelet/pods/a76f088d-58ab-4677-99f3-073e23c2fd13/volumes"
Sep 10 12:31:09.771018 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:09.770991    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"]
Sep 10 12:31:13.942032 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:13.942003    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:31:22.664993 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.664954    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"]
Sep 10 12:31:22.665525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.665508    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a76f088d-58ab-4677-99f3-073e23c2fd13" containerName="main"
Sep 10 12:31:22.665589 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.665528    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a76f088d-58ab-4677-99f3-073e23c2fd13" containerName="main"
Sep 10 12:31:22.665589 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.665549    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" containerName="payload-pre-processing"
Sep 10 12:31:22.665589 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.665558    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" containerName="payload-pre-processing"
Sep 10 12:31:22.665697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.665665    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="903d25fa-3bf4-4c6a-ae26-2ac70b2cda48" containerName="payload-pre-processing"
Sep 10 12:31:22.665697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.665676    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a76f088d-58ab-4677-99f3-073e23c2fd13" containerName="main"
Sep 10 12:31:22.668931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.668913    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.672137 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.672113    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"sub-test-model-18c2d2-kserve-self-signed-certs\""
Sep 10 12:31:22.682475 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.682451    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"]
Sep 10 12:31:22.767947 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.767915    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-home\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.768075 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.767971    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-dshm\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.768118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.768080    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g994v\" (UniqueName: \"kubernetes.io/projected/320dcc71-e348-4b3e-9b54-d072549ee9d2-kube-api-access-g994v\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.768118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.768111    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320dcc71-e348-4b3e-9b54-d072549ee9d2-tls-certs\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.768199 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.768143    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-tmp-dir\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.768199 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.768182    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-model-cache\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.868823 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.868794    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g994v\" (UniqueName: \"kubernetes.io/projected/320dcc71-e348-4b3e-9b54-d072549ee9d2-kube-api-access-g994v\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.868983 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.868831    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320dcc71-e348-4b3e-9b54-d072549ee9d2-tls-certs\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.868983 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.868878    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-tmp-dir\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.868983 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.868906    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-model-cache\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.868983 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.868956    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-home\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.869175 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.869019    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-dshm\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.869272 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.869254    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-tmp-dir\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.869363 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.869340    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-model-cache\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.869456 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.869388    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-home\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.871302 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.871284    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-dshm\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.871489 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.871471    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320dcc71-e348-4b3e-9b54-d072549ee9d2-tls-certs\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.881581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.881550    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g994v\" (UniqueName: \"kubernetes.io/projected/320dcc71-e348-4b3e-9b54-d072549ee9d2-kube-api-access-g994v\") pod \"sub-test-model-18c2d2-kserve-7787dfbb67-ncf77\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:22.978422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:22.978367    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:23.266943 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:23.266872    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"]
Sep 10 12:31:23.275799 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:31:23.275773    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod320dcc71_e348_4b3e_9b54_d072549ee9d2.slice/crio-c8c5f0b2e6c0bfb29c8e2c32061f3201e029b9d49b3c067074c7fd47d8bc85a8 WatchSource:0}: Error finding container c8c5f0b2e6c0bfb29c8e2c32061f3201e029b9d49b3c067074c7fd47d8bc85a8: Status 404 returned error can't find the container with id c8c5f0b2e6c0bfb29c8e2c32061f3201e029b9d49b3c067074c7fd47d8bc85a8
Sep 10 12:31:24.286635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:24.286590    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77" event={"ID":"320dcc71-e348-4b3e-9b54-d072549ee9d2","Type":"ContainerStarted","Data":"730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d"}
Sep 10 12:31:24.287059 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:24.286646    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77" event={"ID":"320dcc71-e348-4b3e-9b54-d072549ee9d2","Type":"ContainerStarted","Data":"c8c5f0b2e6c0bfb29c8e2c32061f3201e029b9d49b3c067074c7fd47d8bc85a8"}
Sep 10 12:31:24.309363 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:24.309315    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77" podStartSLOduration=2.309299146 podStartE2EDuration="2.309299146s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:31:24.307795606 +0000 UTC m=+1295.504311309" watchObservedRunningTime="2026-09-10 12:31:24.309299146 +0000 UTC m=+1295.505814846"
Sep 10 12:31:24.790589 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:24.790545    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67" podUID="88646d2a-1473-4e23-8950-d18e44ce7429" containerName="main" containerID="cri-o://79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0" gracePeriod=45
Sep 10 12:31:25.039964 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.039942    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:31:25.089044 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.088980    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dznwj\" (UniqueName: \"kubernetes.io/projected/88646d2a-1473-4e23-8950-d18e44ce7429-kube-api-access-dznwj\") pod \"88646d2a-1473-4e23-8950-d18e44ce7429\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") "
Sep 10 12:31:25.089044 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089018    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-dshm\") pod \"88646d2a-1473-4e23-8950-d18e44ce7429\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") "
Sep 10 12:31:25.089213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089052    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-model-cache\") pod \"88646d2a-1473-4e23-8950-d18e44ce7429\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") "
Sep 10 12:31:25.089213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089077    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-tmp-dir\") pod \"88646d2a-1473-4e23-8950-d18e44ce7429\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") "
Sep 10 12:31:25.089213 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089136    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-home\") pod \"88646d2a-1473-4e23-8950-d18e44ce7429\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") "
Sep 10 12:31:25.089381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089212    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88646d2a-1473-4e23-8950-d18e44ce7429-tls-certs\") pod \"88646d2a-1473-4e23-8950-d18e44ce7429\" (UID: \"88646d2a-1473-4e23-8950-d18e44ce7429\") "
Sep 10 12:31:25.089381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089370    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-model-cache" (OuterVolumeSpecName: "model-cache") pod "88646d2a-1473-4e23-8950-d18e44ce7429" (UID: "88646d2a-1473-4e23-8950-d18e44ce7429"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:25.089507 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089450    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88646d2a-1473-4e23-8950-d18e44ce7429" (UID: "88646d2a-1473-4e23-8950-d18e44ce7429"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:25.089507 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089494    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-home" (OuterVolumeSpecName: "home") pod "88646d2a-1473-4e23-8950-d18e44ce7429" (UID: "88646d2a-1473-4e23-8950-d18e44ce7429"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:25.089674 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089650    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:25.089877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089805    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:25.089948 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.089882    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:25.091382 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.091353    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-dshm" (OuterVolumeSpecName: "dshm") pod "88646d2a-1473-4e23-8950-d18e44ce7429" (UID: "88646d2a-1473-4e23-8950-d18e44ce7429"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:25.091489 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.091467    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88646d2a-1473-4e23-8950-d18e44ce7429-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88646d2a-1473-4e23-8950-d18e44ce7429" (UID: "88646d2a-1473-4e23-8950-d18e44ce7429"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:31:25.091652 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.091635    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88646d2a-1473-4e23-8950-d18e44ce7429-kube-api-access-dznwj" (OuterVolumeSpecName: "kube-api-access-dznwj") pod "88646d2a-1473-4e23-8950-d18e44ce7429" (UID: "88646d2a-1473-4e23-8950-d18e44ce7429"). InnerVolumeSpecName "kube-api-access-dznwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:31:25.190905 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.190874    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88646d2a-1473-4e23-8950-d18e44ce7429-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:25.190905 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.190906    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dznwj\" (UniqueName: \"kubernetes.io/projected/88646d2a-1473-4e23-8950-d18e44ce7429-kube-api-access-dznwj\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:25.190905 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.190916    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88646d2a-1473-4e23-8950-d18e44ce7429-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:25.293100 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.293068    2552 generic.go:356] "Generic (PLEG): container finished" podID="88646d2a-1473-4e23-8950-d18e44ce7429" containerID="79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0" exitCode=0
Sep 10 12:31:25.293422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.293105    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"
Sep 10 12:31:25.293422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.293144    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67" event={"ID":"88646d2a-1473-4e23-8950-d18e44ce7429","Type":"ContainerDied","Data":"79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0"}
Sep 10 12:31:25.293422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.293183    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67" event={"ID":"88646d2a-1473-4e23-8950-d18e44ce7429","Type":"ContainerDied","Data":"6e8c5336b2d2bde5a86f4d2b6cdb20d392f3497e78c2bdd285d5efcad7a9dedb"}
Sep 10 12:31:25.293422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.293204    2552 scope.go:117] "RemoveContainer" containerID="79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0"
Sep 10 12:31:25.302578 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.302562    2552 scope.go:117] "RemoveContainer" containerID="79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0"
Sep 10 12:31:25.302827 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:31:25.302806    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0\": container with ID starting with 79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0 not found: ID does not exist" containerID="79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0"
Sep 10 12:31:25.302869 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.302840    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0"} err="failed to get container status \"79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0\": rpc error: code = NotFound desc = could not find container \"79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0\": container with ID starting with 79cbda50270b6a41c046d7d6336906bc6d89ba0157e10558f21f282074f3cad0 not found: ID does not exist"
Sep 10 12:31:25.312815 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.312795    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"]
Sep 10 12:31:25.316348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.316328    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/auth-test-model-18c2d2-kserve-7677946555-vzx67"]
Sep 10 12:31:25.860596 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:25.860561    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:31:27.243518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:27.243480    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88646d2a-1473-4e23-8950-d18e44ce7429" path="/var/lib/kubelet/pods/88646d2a-1473-4e23-8950-d18e44ce7429/volumes"
Sep 10 12:31:32.979471 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:32.979440    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:32.979860 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:32.979486    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:32.992527 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:32.992504    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:33.345375 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:33.345304    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:31:34.664877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:34.664842    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"]
Sep 10 12:31:34.665700 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:34.665665    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy" containerID="cri-o://fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505" gracePeriod=30
Sep 10 12:31:39.922923 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:39.922897    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:31:40.017051 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.016989    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-podinfo\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017051 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017036    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-h6btk\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-kube-api-access-h6btk\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017193 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017066    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-socket\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017279 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017243    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-credential-socket\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017314    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-certs\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017358    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istiod-ca-cert\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017315    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:40.017452 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017397    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-envoy\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017573 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017502    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-data\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017573 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017512    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:40.017573 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017540    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-token\") pod \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\" (UID: \"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79\") "
Sep 10 12:31:40.017573 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017553    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:40.017795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017669    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-data" (OuterVolumeSpecName: "istio-data") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:40.017795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.017766    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:31:40.018040 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.018019    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.018288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.018044    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.018288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.018058    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.018288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.018071    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.018288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.018086    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.019361 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.019339    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:31:40.019494 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.019469    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-kube-api-access-h6btk" (OuterVolumeSpecName: "kube-api-access-h6btk") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "kube-api-access-h6btk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:31:40.019564 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.019539    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:31:40.019736 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.019717    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-token" (OuterVolumeSpecName: "istio-token") pod "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" (UID: "d77ed0b6-db56-4fb3-8585-42c3d3e2cc79"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:31:40.119237 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.119216    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.119237 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.119236    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.119361 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.119245    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.119361 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.119254    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h6btk\" (UniqueName: \"kubernetes.io/projected/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79-kube-api-access-h6btk\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:31:40.159175 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.159140    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:31:40.371942 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.371919    2552 generic.go:356] "Generic (PLEG): container finished" podID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerID="fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505" exitCode=0
Sep 10 12:31:40.372054 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.371966    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"
Sep 10 12:31:40.372054 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.372002    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" event={"ID":"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79","Type":"ContainerDied","Data":"fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505"}
Sep 10 12:31:40.372125 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.372055    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv" event={"ID":"d77ed0b6-db56-4fb3-8585-42c3d3e2cc79","Type":"ContainerDied","Data":"2b45983eb907e47bdb9ac93568c0a3c60ba2f2bef8439c85fa0a54d7003a05ec"}
Sep 10 12:31:40.372125 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.372076    2552 scope.go:117] "RemoveContainer" containerID="fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505"
Sep 10 12:31:40.381322 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.381305    2552 scope.go:117] "RemoveContainer" containerID="d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa"
Sep 10 12:31:40.389414 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.389397    2552 scope.go:117] "RemoveContainer" containerID="fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505"
Sep 10 12:31:40.389704 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:31:40.389687    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505\": container with ID starting with fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505 not found: ID does not exist" containerID="fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505"
Sep 10 12:31:40.389773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.389710    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505"} err="failed to get container status \"fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505\": rpc error: code = NotFound desc = could not find container \"fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505\": container with ID starting with fcf0a0a7fd876eb55a9bb44444f6bdf07909ff34332e8550dfb17ffbe791f505 not found: ID does not exist"
Sep 10 12:31:40.389773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.389724    2552 scope.go:117] "RemoveContainer" containerID="d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa"
Sep 10 12:31:40.389962 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:31:40.389948    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa\": container with ID starting with d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa not found: ID does not exist" containerID="d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa"
Sep 10 12:31:40.390003 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.389966    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa"} err="failed to get container status \"d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa\": rpc error: code = NotFound desc = could not find container \"d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa\": container with ID starting with d71e77513e4ddb5f47d4b7803a841d12dd5d8550a90daba82b8ae1d1700bd3fa not found: ID does not exist"
Sep 10 12:31:40.405309 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.405286    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"]
Sep 10 12:31:40.412221 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:40.412200    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-156d6252-openshift-default-c88fc69fb-r2jcv"]
Sep 10 12:31:41.243232 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:41.243194    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" path="/var/lib/kubelet/pods/d77ed0b6-db56-4fb3-8585-42c3d3e2cc79/volumes"
Sep 10 12:31:49.418104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:49.418077    2552 scope.go:117] "RemoveContainer" containerID="80bb50b2f0d9a7af99cd8ad254c52a0c57a596b8d5449aac09fe9fc97816e935"
Sep 10 12:31:57.152677 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:57.152639    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:31:59.490047 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490018    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"]
Sep 10 12:31:59.490404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490351    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy"
Sep 10 12:31:59.490404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490361    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy"
Sep 10 12:31:59.490404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490373    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy"
Sep 10 12:31:59.490404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490379    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy"
Sep 10 12:31:59.490404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490393    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88646d2a-1473-4e23-8950-d18e44ce7429" containerName="main"
Sep 10 12:31:59.490404 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490398    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="88646d2a-1473-4e23-8950-d18e44ce7429" containerName="main"
Sep 10 12:31:59.490636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490475    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="88646d2a-1473-4e23-8950-d18e44ce7429" containerName="main"
Sep 10 12:31:59.490636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490483    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy"
Sep 10 12:31:59.490636 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.490489    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d77ed0b6-db56-4fb3-8585-42c3d3e2cc79" containerName="istio-proxy"
Sep 10 12:31:59.494840 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.494822    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.502267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.502247    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"openshift-service-ca.crt\""
Sep 10 12:31:59.502377 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.502287    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"default-dockercfg-rfdfk\""
Sep 10 12:31:59.502377 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.502369    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"kube-root-ca.crt\""
Sep 10 12:31:59.502524 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.502398    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"sub-test-model-f89a79-kserve-self-signed-certs\""
Sep 10 12:31:59.508625 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.508604    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"]
Sep 10 12:31:59.568518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.568495    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lskqw\" (UniqueName: \"kubernetes.io/projected/f9d781fe-5816-4b44-8819-2b0e77c9fb19-kube-api-access-lskqw\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.568650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.568552    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-home\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.568650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.568570    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tmp-dir\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.568769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.568657    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-dshm\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.568769 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.568698    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-model-cache\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.568848 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.568771    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tls-certs\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.669525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.669494    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-dshm\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.669704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.669539    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-model-cache\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.669704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.669588    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tls-certs\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.669704 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.669632    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lskqw\" (UniqueName: \"kubernetes.io/projected/f9d781fe-5816-4b44-8819-2b0e77c9fb19-kube-api-access-lskqw\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.669872 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.669714    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-home\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.669872 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.669739    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tmp-dir\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.670019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.669987    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-model-cache\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.670019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.670016    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-home\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.670134 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.670079    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tmp-dir\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.671875 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.671848    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-dshm\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.672103 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.672086    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tls-certs\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.680195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.680175    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lskqw\" (UniqueName: \"kubernetes.io/projected/f9d781fe-5816-4b44-8819-2b0e77c9fb19-kube-api-access-lskqw\") pod \"sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:31:59.803853 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:31:59.803790    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:32:00.086938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:00.086912    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"]
Sep 10 12:32:00.089080 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:32:00.089050    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf9d781fe_5816_4b44_8819_2b0e77c9fb19.slice/crio-18dcbd02ca5325e9f66a2f95141ce783661789a0191da225d61dc85d3b545bb8 WatchSource:0}: Error finding container 18dcbd02ca5325e9f66a2f95141ce783661789a0191da225d61dc85d3b545bb8: Status 404 returned error can't find the container with id 18dcbd02ca5325e9f66a2f95141ce783661789a0191da225d61dc85d3b545bb8
Sep 10 12:32:00.477386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:00.477352    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm" event={"ID":"f9d781fe-5816-4b44-8819-2b0e77c9fb19","Type":"ContainerStarted","Data":"705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db"}
Sep 10 12:32:00.477386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:00.477390    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm" event={"ID":"f9d781fe-5816-4b44-8819-2b0e77c9fb19","Type":"ContainerStarted","Data":"18dcbd02ca5325e9f66a2f95141ce783661789a0191da225d61dc85d3b545bb8"}
Sep 10 12:32:00.501554 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:00.501512    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm" podStartSLOduration=1.501499319 podStartE2EDuration="1.501499319s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:31:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:00.501067526 +0000 UTC m=+1331.697583225" watchObservedRunningTime="2026-09-10 12:32:00.501499319 +0000 UTC m=+1331.698015045"
Sep 10 12:32:05.267465 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:05.267419    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:32:09.804372 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:09.804343    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:32:09.804786 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:09.804489    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:32:09.817118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:09.817096    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:32:10.535446 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:10.535360    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:32:24.060506 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:24.060469    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:32:36.666491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.666451    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"]
Sep 10 12:32:36.670228 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.670206    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:36.673462 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.673445    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-5ff68c\""
Sep 10 12:32:36.686479 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.686413    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"]
Sep 10 12:32:36.795336 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.795295    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grrcp\" (UniqueName: \"kubernetes.io/projected/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-kube-api-access-grrcp\") pod \"payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:36.795555 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.795511    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:36.896542 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.896496    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grrcp\" (UniqueName: \"kubernetes.io/projected/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-kube-api-access-grrcp\") pod \"payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:36.896747 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.896595    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:36.897291 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.897269    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:36.906410 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.906376    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-grrcp\" (UniqueName: \"kubernetes.io/projected/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-kube-api-access-grrcp\") pod \"payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:36.981265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:36.981179    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:37.258754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:37.258729    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"]
Sep 10 12:32:37.260772 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:32:37.260743    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73aefad6_a5ee_4051_87d0_7d6e8e0dc8d6.slice/crio-49bc3ce0b3f3d49e03d38eff34e2e3f8d6ba9e557d20224f13d343307b298c24 WatchSource:0}: Error finding container 49bc3ce0b3f3d49e03d38eff34e2e3f8d6ba9e557d20224f13d343307b298c24: Status 404 returned error can't find the container with id 49bc3ce0b3f3d49e03d38eff34e2e3f8d6ba9e557d20224f13d343307b298c24
Sep 10 12:32:37.658571 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:37.658533    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9" event={"ID":"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6","Type":"ContainerStarted","Data":"0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014"}
Sep 10 12:32:37.658768 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:37.658590    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9" event={"ID":"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6","Type":"ContainerStarted","Data":"49bc3ce0b3f3d49e03d38eff34e2e3f8d6ba9e557d20224f13d343307b298c24"}
Sep 10 12:32:37.658768 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:37.658629    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:37.681149 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:37.681099    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9" podStartSLOduration=1.68108335 podStartE2EDuration="1.68108335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:32:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:32:37.679535077 +0000 UTC m=+1368.876050776" watchObservedRunningTime="2026-09-10 12:32:37.68108335 +0000 UTC m=+1368.877599051"
Sep 10 12:32:40.663815 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:40.663780    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:32:44.754591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:44.754557    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7600-51200d"]
Sep 10 12:32:44.758054 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:44.758036    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:44.760897 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:44.760876    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-grlsn\""
Sep 10 12:32:44.766968 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:44.766943    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7600-51200d"]
Sep 10 12:32:44.775167 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:44.775136    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5nhw\" (UniqueName: \"kubernetes.io/projected/a312c3f7-9db0-44e1-8c4e-a64898711f91-kube-api-access-z5nhw\") pod \"mt-curl-7600-51200d\" (UID: \"a312c3f7-9db0-44e1-8c4e-a64898711f91\") " pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:44.876174 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:44.876143    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5nhw\" (UniqueName: \"kubernetes.io/projected/a312c3f7-9db0-44e1-8c4e-a64898711f91-kube-api-access-z5nhw\") pod \"mt-curl-7600-51200d\" (UID: \"a312c3f7-9db0-44e1-8c4e-a64898711f91\") " pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:44.885250 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:44.885217    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5nhw\" (UniqueName: \"kubernetes.io/projected/a312c3f7-9db0-44e1-8c4e-a64898711f91-kube-api-access-z5nhw\") pod \"mt-curl-7600-51200d\" (UID: \"a312c3f7-9db0-44e1-8c4e-a64898711f91\") " pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:45.068519 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:45.068420    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:45.346957 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:45.346923    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7600-51200d"]
Sep 10 12:32:45.348592 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:32:45.348564    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda312c3f7_9db0_44e1_8c4e_a64898711f91.slice/crio-6116118fb45f334a071e011677b8cb99dcbde94d0b5a77debf466eedc3466fc3 WatchSource:0}: Error finding container 6116118fb45f334a071e011677b8cb99dcbde94d0b5a77debf466eedc3466fc3: Status 404 returned error can't find the container with id 6116118fb45f334a071e011677b8cb99dcbde94d0b5a77debf466eedc3466fc3
Sep 10 12:32:45.705800 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:45.705760    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7600-51200d" event={"ID":"a312c3f7-9db0-44e1-8c4e-a64898711f91","Type":"ContainerStarted","Data":"6116118fb45f334a071e011677b8cb99dcbde94d0b5a77debf466eedc3466fc3"}
Sep 10 12:32:46.712596 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:46.712563    2552 generic.go:356] "Generic (PLEG): container finished" podID="a312c3f7-9db0-44e1-8c4e-a64898711f91" containerID="5c1da28b8df3ea2490d6d5012c547594ac59043f01c4f2e7213148a6cf58e18c" exitCode=0
Sep 10 12:32:46.712975 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:46.712653    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7600-51200d" event={"ID":"a312c3f7-9db0-44e1-8c4e-a64898711f91","Type":"ContainerDied","Data":"5c1da28b8df3ea2490d6d5012c547594ac59043f01c4f2e7213148a6cf58e18c"}
Sep 10 12:32:47.858766 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:47.858738    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:47.900079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:47.900048    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7600-51200d_a312c3f7-9db0-44e1-8c4e-a64898711f91/mt-curl-7600-51200d/0.log"
Sep 10 12:32:47.910048 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:47.910013    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z5nhw\" (UniqueName: \"kubernetes.io/projected/a312c3f7-9db0-44e1-8c4e-a64898711f91-kube-api-access-z5nhw\") pod \"a312c3f7-9db0-44e1-8c4e-a64898711f91\" (UID: \"a312c3f7-9db0-44e1-8c4e-a64898711f91\") "
Sep 10 12:32:47.912503 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:47.912470    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a312c3f7-9db0-44e1-8c4e-a64898711f91-kube-api-access-z5nhw" (OuterVolumeSpecName: "kube-api-access-z5nhw") pod "a312c3f7-9db0-44e1-8c4e-a64898711f91" (UID: "a312c3f7-9db0-44e1-8c4e-a64898711f91"). InnerVolumeSpecName "kube-api-access-z5nhw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:32:47.926732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:47.926698    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7600-51200d"]
Sep 10 12:32:47.929846 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:47.929815    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7600-51200d"]
Sep 10 12:32:48.011845 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.011741    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5nhw\" (UniqueName: \"kubernetes.io/projected/a312c3f7-9db0-44e1-8c4e-a64898711f91-kube-api-access-z5nhw\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:32:48.045642 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.045606    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/mt-curl-7600-e18a99"]
Sep 10 12:32:48.046023 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.046008    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a312c3f7-9db0-44e1-8c4e-a64898711f91" containerName="mt-curl-7600-51200d"
Sep 10 12:32:48.046069 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.046026    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a312c3f7-9db0-44e1-8c4e-a64898711f91" containerName="mt-curl-7600-51200d"
Sep 10 12:32:48.046104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.046099    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a312c3f7-9db0-44e1-8c4e-a64898711f91" containerName="mt-curl-7600-51200d"
Sep 10 12:32:48.049104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.049088    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-e18a99"
Sep 10 12:32:48.066024 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.065989    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7600-e18a99"]
Sep 10 12:32:48.112545 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.112501    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hcz6\" (UniqueName: \"kubernetes.io/projected/8d271e53-d653-4599-aaa0-0453f27d7b24-kube-api-access-4hcz6\") pod \"mt-curl-7600-e18a99\" (UID: \"8d271e53-d653-4599-aaa0-0453f27d7b24\") " pod="opendatahub/mt-curl-7600-e18a99"
Sep 10 12:32:48.213862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.213814    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hcz6\" (UniqueName: \"kubernetes.io/projected/8d271e53-d653-4599-aaa0-0453f27d7b24-kube-api-access-4hcz6\") pod \"mt-curl-7600-e18a99\" (UID: \"8d271e53-d653-4599-aaa0-0453f27d7b24\") " pod="opendatahub/mt-curl-7600-e18a99"
Sep 10 12:32:48.231595 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.231557    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hcz6\" (UniqueName: \"kubernetes.io/projected/8d271e53-d653-4599-aaa0-0453f27d7b24-kube-api-access-4hcz6\") pod \"mt-curl-7600-e18a99\" (UID: \"8d271e53-d653-4599-aaa0-0453f27d7b24\") " pod="opendatahub/mt-curl-7600-e18a99"
Sep 10 12:32:48.358069 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.357976    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-e18a99"
Sep 10 12:32:48.634385 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.634358    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/mt-curl-7600-e18a99"]
Sep 10 12:32:48.636703 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:32:48.636678    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8d271e53_d653_4599_aaa0_0453f27d7b24.slice/crio-7dd177235abccd55260cc258afbf5b4d63d39f5db72735b5307e0858d43c7786 WatchSource:0}: Error finding container 7dd177235abccd55260cc258afbf5b4d63d39f5db72735b5307e0858d43c7786: Status 404 returned error can't find the container with id 7dd177235abccd55260cc258afbf5b4d63d39f5db72735b5307e0858d43c7786
Sep 10 12:32:48.666018 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.666000    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:32:48.724315 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.724263    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7600-e18a99" event={"ID":"8d271e53-d653-4599-aaa0-0453f27d7b24","Type":"ContainerStarted","Data":"7dd177235abccd55260cc258afbf5b4d63d39f5db72735b5307e0858d43c7786"}
Sep 10 12:32:48.725674 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.725648    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:48.725775 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:48.725701    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6116118fb45f334a071e011677b8cb99dcbde94d0b5a77debf466eedc3466fc3"
Sep 10 12:32:48.737691 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:32:48.737667    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7600-51200d\" is forbidden: User \"system:node:ip-10-0-129-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-68.ec2.internal' and this object" podUID="a312c3f7-9db0-44e1-8c4e-a64898711f91" pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:49.241115 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:32:49.241053    2552 status_manager.go:1018] "Failed to get status for pod" err="pods \"mt-curl-7600-51200d\" is forbidden: User \"system:node:ip-10-0-129-68.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-68.ec2.internal' and this object" podUID="a312c3f7-9db0-44e1-8c4e-a64898711f91" pod="opendatahub/mt-curl-7600-51200d"
Sep 10 12:32:49.243807 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:49.243781    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a312c3f7-9db0-44e1-8c4e-a64898711f91" path="/var/lib/kubelet/pods/a312c3f7-9db0-44e1-8c4e-a64898711f91/volumes"
Sep 10 12:32:49.447670 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:49.447645    2552 scope.go:117] "RemoveContainer" containerID="fc380757a5e3c33d1793d8a08a31183bf4e19031d0bbdd0c9a5e1b95f281f656"
Sep 10 12:32:49.731649 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:49.731624    2552 generic.go:356] "Generic (PLEG): container finished" podID="8d271e53-d653-4599-aaa0-0453f27d7b24" containerID="e3c5cf47481a720e4764cc7dca377a5ff1fd97fca8b990c3978533e66238cf98" exitCode=0
Sep 10 12:32:49.731765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:49.731708    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/mt-curl-7600-e18a99" event={"ID":"8d271e53-d653-4599-aaa0-0453f27d7b24","Type":"ContainerDied","Data":"e3c5cf47481a720e4764cc7dca377a5ff1fd97fca8b990c3978533e66238cf98"}
Sep 10 12:32:50.886500 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:50.886472    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-e18a99"
Sep 10 12:32:50.918572 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:50.918546    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_mt-curl-7600-e18a99_8d271e53-d653-4599-aaa0-0453f27d7b24/mt-curl-7600-e18a99/0.log"
Sep 10 12:32:50.936561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:50.936531    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4hcz6\" (UniqueName: \"kubernetes.io/projected/8d271e53-d653-4599-aaa0-0453f27d7b24-kube-api-access-4hcz6\") pod \"8d271e53-d653-4599-aaa0-0453f27d7b24\" (UID: \"8d271e53-d653-4599-aaa0-0453f27d7b24\") "
Sep 10 12:32:50.938731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:50.938697    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8d271e53-d653-4599-aaa0-0453f27d7b24-kube-api-access-4hcz6" (OuterVolumeSpecName: "kube-api-access-4hcz6") pod "8d271e53-d653-4599-aaa0-0453f27d7b24" (UID: "8d271e53-d653-4599-aaa0-0453f27d7b24"). InnerVolumeSpecName "kube-api-access-4hcz6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:32:50.949268 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:50.949246    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/mt-curl-7600-e18a99"]
Sep 10 12:32:50.951575 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:50.951553    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/mt-curl-7600-e18a99"]
Sep 10 12:32:51.037768 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:51.037677    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hcz6\" (UniqueName: \"kubernetes.io/projected/8d271e53-d653-4599-aaa0-0453f27d7b24-kube-api-access-4hcz6\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:32:51.242512 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:51.242477    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8d271e53-d653-4599-aaa0-0453f27d7b24" path="/var/lib/kubelet/pods/8d271e53-d653-4599-aaa0-0453f27d7b24/volumes"
Sep 10 12:32:51.743265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:51.743233    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/mt-curl-7600-e18a99"
Sep 10 12:32:51.743420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:51.743269    2552 scope.go:117] "RemoveContainer" containerID="e3c5cf47481a720e4764cc7dca377a5ff1fd97fca8b990c3978533e66238cf98"
Sep 10 12:32:52.961646 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:52.961616    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"]
Sep 10 12:32:53.753457 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:53.753411    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"]
Sep 10 12:32:58.452237 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:32:58.452205    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:33:00.248863 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.248837    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"]
Sep 10 12:33:00.249330 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.249308    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8d271e53-d653-4599-aaa0-0453f27d7b24" containerName="mt-curl-7600-e18a99"
Sep 10 12:33:00.249330 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.249328    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8d271e53-d653-4599-aaa0-0453f27d7b24" containerName="mt-curl-7600-e18a99"
Sep 10 12:33:00.249536 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.249409    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8d271e53-d653-4599-aaa0-0453f27d7b24" containerName="mt-curl-7600-e18a99"
Sep 10 12:33:00.253929 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.253909    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.256841 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.256822    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-da7b8f-openshift-default-dockercfg-hstbr\""
Sep 10 12:33:00.268280 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.268257    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"]
Sep 10 12:33:00.318004 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.317977    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-envoy\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318133 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318045    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972fe1af-6eae-4789-8e38-6bc959916e86-istiod-ca-cert\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318133 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318075    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-socket\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318144    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-istio-token\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318191    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-credential-socket\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318259 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318256    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-certs\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318388 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318294    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-data\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318388 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318371    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5jm8z\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-kube-api-access-5jm8z\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.318488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.318410    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972fe1af-6eae-4789-8e38-6bc959916e86-istio-podinfo\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.419703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.419676    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-certs\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.419863 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.419714    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-data\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.419863 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.419757    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5jm8z\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-kube-api-access-5jm8z\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.419863 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.419785    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972fe1af-6eae-4789-8e38-6bc959916e86-istio-podinfo\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.419863 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.419817    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-envoy\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420088 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420025    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972fe1af-6eae-4789-8e38-6bc959916e86-istiod-ca-cert\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420088 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420066    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-socket\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420104    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-certs\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420118    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-istio-token\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420146    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-data\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420166    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-credential-socket\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420447 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420402    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-socket\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420523 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420458    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-credential-socket\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.420798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.420775    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972fe1af-6eae-4789-8e38-6bc959916e86-istiod-ca-cert\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.422132 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.422114    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-envoy\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.422341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.422325    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972fe1af-6eae-4789-8e38-6bc959916e86-istio-podinfo\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.429711 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.429690    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5jm8z\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-kube-api-access-5jm8z\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.431674 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.431654    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-istio-token\") pod \"e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") " pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.563127 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.563074    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:00.850667 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.850641    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"]
Sep 10 12:33:00.851775 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:33:00.851755    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod972fe1af_6eae_4789_8e38_6bc959916e86.slice/crio-c0cccf13a0c7cebe4ebd815df77419dafbdb41215f58712943b6d6f1b0412be5 WatchSource:0}: Error finding container c0cccf13a0c7cebe4ebd815df77419dafbdb41215f58712943b6d6f1b0412be5: Status 404 returned error can't find the container with id c0cccf13a0c7cebe4ebd815df77419dafbdb41215f58712943b6d6f1b0412be5
Sep 10 12:33:00.853814 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.853777    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:33:00.853915 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.853848    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:33:00.853915 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:00.853877    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:33:01.795890 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:01.795836    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" event={"ID":"972fe1af-6eae-4789-8e38-6bc959916e86","Type":"ContainerStarted","Data":"ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35"}
Sep 10 12:33:01.796265 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:01.795909    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" event={"ID":"972fe1af-6eae-4789-8e38-6bc959916e86","Type":"ContainerStarted","Data":"c0cccf13a0c7cebe4ebd815df77419dafbdb41215f58712943b6d6f1b0412be5"}
Sep 10 12:33:01.822312 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:01.822268    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" podStartSLOduration=1.822253684 podStartE2EDuration="1.822253684s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:33:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:33:01.821338498 +0000 UTC m=+1393.017854199" watchObservedRunningTime="2026-09-10 12:33:01.822253684 +0000 UTC m=+1393.018769799"
Sep 10 12:33:02.564099 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:02.564064    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:02.568865 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:02.568844    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:02.800986 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:02.800959    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:02.801979 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:02.801957    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:07.980933 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:07.980893    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77" podUID="320dcc71-e348-4b3e-9b54-d072549ee9d2" containerName="main" containerID="cri-o://730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d" gracePeriod=45
Sep 10 12:33:08.238514 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.238451    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:33:08.393731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.393701    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-tmp-dir\") pod \"320dcc71-e348-4b3e-9b54-d072549ee9d2\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") "
Sep 10 12:33:08.393888 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.393761    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320dcc71-e348-4b3e-9b54-d072549ee9d2-tls-certs\") pod \"320dcc71-e348-4b3e-9b54-d072549ee9d2\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") "
Sep 10 12:33:08.393888 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.393794    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-dshm\") pod \"320dcc71-e348-4b3e-9b54-d072549ee9d2\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") "
Sep 10 12:33:08.393888 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.393815    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-model-cache\") pod \"320dcc71-e348-4b3e-9b54-d072549ee9d2\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") "
Sep 10 12:33:08.393888 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.393860    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g994v\" (UniqueName: \"kubernetes.io/projected/320dcc71-e348-4b3e-9b54-d072549ee9d2-kube-api-access-g994v\") pod \"320dcc71-e348-4b3e-9b54-d072549ee9d2\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") "
Sep 10 12:33:08.393888 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.393890    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-home\") pod \"320dcc71-e348-4b3e-9b54-d072549ee9d2\" (UID: \"320dcc71-e348-4b3e-9b54-d072549ee9d2\") "
Sep 10 12:33:08.394180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.394004    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "320dcc71-e348-4b3e-9b54-d072549ee9d2" (UID: "320dcc71-e348-4b3e-9b54-d072549ee9d2"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:08.394180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.394098    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-model-cache" (OuterVolumeSpecName: "model-cache") pod "320dcc71-e348-4b3e-9b54-d072549ee9d2" (UID: "320dcc71-e348-4b3e-9b54-d072549ee9d2"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:08.394307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.394222    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-home" (OuterVolumeSpecName: "home") pod "320dcc71-e348-4b3e-9b54-d072549ee9d2" (UID: "320dcc71-e348-4b3e-9b54-d072549ee9d2"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:08.395036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.395005    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:08.395150 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.395041    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:08.396799 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.396756    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-dshm" (OuterVolumeSpecName: "dshm") pod "320dcc71-e348-4b3e-9b54-d072549ee9d2" (UID: "320dcc71-e348-4b3e-9b54-d072549ee9d2"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:08.403693 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.403666    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/320dcc71-e348-4b3e-9b54-d072549ee9d2-kube-api-access-g994v" (OuterVolumeSpecName: "kube-api-access-g994v") pod "320dcc71-e348-4b3e-9b54-d072549ee9d2" (UID: "320dcc71-e348-4b3e-9b54-d072549ee9d2"). InnerVolumeSpecName "kube-api-access-g994v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:08.404029 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.404007    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/320dcc71-e348-4b3e-9b54-d072549ee9d2-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "320dcc71-e348-4b3e-9b54-d072549ee9d2" (UID: "320dcc71-e348-4b3e-9b54-d072549ee9d2"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:08.496105 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.496043    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g994v\" (UniqueName: \"kubernetes.io/projected/320dcc71-e348-4b3e-9b54-d072549ee9d2-kube-api-access-g994v\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:08.496105 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.496069    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:08.496105 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.496081    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/320dcc71-e348-4b3e-9b54-d072549ee9d2-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:08.496105 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.496088    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/320dcc71-e348-4b3e-9b54-d072549ee9d2-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:08.773669 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.773565    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm" podUID="f9d781fe-5816-4b44-8819-2b0e77c9fb19" containerName="main" containerID="cri-o://705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db" gracePeriod=45
Sep 10 12:33:08.835053 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.835018    2552 generic.go:356] "Generic (PLEG): container finished" podID="320dcc71-e348-4b3e-9b54-d072549ee9d2" containerID="730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d" exitCode=0
Sep 10 12:33:08.835215 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.835074    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"
Sep 10 12:33:08.835215 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.835102    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77" event={"ID":"320dcc71-e348-4b3e-9b54-d072549ee9d2","Type":"ContainerDied","Data":"730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d"}
Sep 10 12:33:08.835215 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.835153    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77" event={"ID":"320dcc71-e348-4b3e-9b54-d072549ee9d2","Type":"ContainerDied","Data":"c8c5f0b2e6c0bfb29c8e2c32061f3201e029b9d49b3c067074c7fd47d8bc85a8"}
Sep 10 12:33:08.835215 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.835176    2552 scope.go:117] "RemoveContainer" containerID="730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d"
Sep 10 12:33:08.860737 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.860710    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"]
Sep 10 12:33:08.867590 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.867569    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/sub-test-model-18c2d2-kserve-7787dfbb67-ncf77"]
Sep 10 12:33:08.884568 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.884546    2552 scope.go:117] "RemoveContainer" containerID="730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d"
Sep 10 12:33:08.884951 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:33:08.884930    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d\": container with ID starting with 730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d not found: ID does not exist" containerID="730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d"
Sep 10 12:33:08.885030 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:08.884962    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d"} err="failed to get container status \"730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d\": rpc error: code = NotFound desc = could not find container \"730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d\": container with ID starting with 730e3d31a6c80a0bd64b3fdade08972c8642273de05379c0b71e25bcf982012d not found: ID does not exist"
Sep 10 12:33:09.030335 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.030276    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:33:09.100809 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.100776    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-model-cache\") pod \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") "
Sep 10 12:33:09.100809 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.100813    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tmp-dir\") pod \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") "
Sep 10 12:33:09.101015 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.100833    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-dshm\") pod \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") "
Sep 10 12:33:09.101015 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.100895    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lskqw\" (UniqueName: \"kubernetes.io/projected/f9d781fe-5816-4b44-8819-2b0e77c9fb19-kube-api-access-lskqw\") pod \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") "
Sep 10 12:33:09.101015 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.100944    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tls-certs\") pod \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") "
Sep 10 12:33:09.101174 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.101027    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-home\") pod \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\" (UID: \"f9d781fe-5816-4b44-8819-2b0e77c9fb19\") "
Sep 10 12:33:09.101174 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.101038    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "f9d781fe-5816-4b44-8819-2b0e77c9fb19" (UID: "f9d781fe-5816-4b44-8819-2b0e77c9fb19"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:09.101174 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.101051    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-model-cache" (OuterVolumeSpecName: "model-cache") pod "f9d781fe-5816-4b44-8819-2b0e77c9fb19" (UID: "f9d781fe-5816-4b44-8819-2b0e77c9fb19"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:09.101326 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.101311    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:09.101377 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.101317    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-home" (OuterVolumeSpecName: "home") pod "f9d781fe-5816-4b44-8819-2b0e77c9fb19" (UID: "f9d781fe-5816-4b44-8819-2b0e77c9fb19"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:09.101377 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.101329    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:09.103231 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.103201    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "f9d781fe-5816-4b44-8819-2b0e77c9fb19" (UID: "f9d781fe-5816-4b44-8819-2b0e77c9fb19"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:33:09.103347 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.103238    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-dshm" (OuterVolumeSpecName: "dshm") pod "f9d781fe-5816-4b44-8819-2b0e77c9fb19" (UID: "f9d781fe-5816-4b44-8819-2b0e77c9fb19"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:33:09.103347 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.103248    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9d781fe-5816-4b44-8819-2b0e77c9fb19-kube-api-access-lskqw" (OuterVolumeSpecName: "kube-api-access-lskqw") pod "f9d781fe-5816-4b44-8819-2b0e77c9fb19" (UID: "f9d781fe-5816-4b44-8819-2b0e77c9fb19"). InnerVolumeSpecName "kube-api-access-lskqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:09.202318 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.202293    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:09.202318 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.202312    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f9d781fe-5816-4b44-8819-2b0e77c9fb19-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:09.202318 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.202321    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lskqw\" (UniqueName: \"kubernetes.io/projected/f9d781fe-5816-4b44-8819-2b0e77c9fb19-kube-api-access-lskqw\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:09.202533 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.202329    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f9d781fe-5816-4b44-8819-2b0e77c9fb19-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:09.242143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.242115    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="320dcc71-e348-4b3e-9b54-d072549ee9d2" path="/var/lib/kubelet/pods/320dcc71-e348-4b3e-9b54-d072549ee9d2/volumes"
Sep 10 12:33:09.840669 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.840637    2552 generic.go:356] "Generic (PLEG): container finished" podID="f9d781fe-5816-4b44-8819-2b0e77c9fb19" containerID="705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db" exitCode=0
Sep 10 12:33:09.840839 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.840679    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"
Sep 10 12:33:09.840839 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.840723    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm" event={"ID":"f9d781fe-5816-4b44-8819-2b0e77c9fb19","Type":"ContainerDied","Data":"705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db"}
Sep 10 12:33:09.840839 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.840761    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm" event={"ID":"f9d781fe-5816-4b44-8819-2b0e77c9fb19","Type":"ContainerDied","Data":"18dcbd02ca5325e9f66a2f95141ce783661789a0191da225d61dc85d3b545bb8"}
Sep 10 12:33:09.840839 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.840778    2552 scope.go:117] "RemoveContainer" containerID="705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db"
Sep 10 12:33:09.849184 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.849169    2552 scope.go:117] "RemoveContainer" containerID="705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db"
Sep 10 12:33:09.849488 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:33:09.849469    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db\": container with ID starting with 705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db not found: ID does not exist" containerID="705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db"
Sep 10 12:33:09.849553 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.849496    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db"} err="failed to get container status \"705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db\": rpc error: code = NotFound desc = could not find container \"705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db\": container with ID starting with 705e9a968ad5b0e9ae2f7d316a5d0a197aadabd97e246f3faddd77b6192e77db not found: ID does not exist"
Sep 10 12:33:09.863138 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.863116    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"]
Sep 10 12:33:09.868141 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:09.868122    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/sub-test-model-f89a79-kserve-6f5654fd9f-7t7tm"]
Sep 10 12:33:11.242099 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:11.242071    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f9d781fe-5816-4b44-8819-2b0e77c9fb19" path="/var/lib/kubelet/pods/f9d781fe-5816-4b44-8819-2b0e77c9fb19/volumes"
Sep 10 12:33:11.560956 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:11.560882    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:33:15.594915 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.594886    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"]
Sep 10 12:33:15.595273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.595238    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="320dcc71-e348-4b3e-9b54-d072549ee9d2" containerName="main"
Sep 10 12:33:15.595273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.595248    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="320dcc71-e348-4b3e-9b54-d072549ee9d2" containerName="main"
Sep 10 12:33:15.595273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.595263    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f9d781fe-5816-4b44-8819-2b0e77c9fb19" containerName="main"
Sep 10 12:33:15.595273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.595269    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="f9d781fe-5816-4b44-8819-2b0e77c9fb19" containerName="main"
Sep 10 12:33:15.595409 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.595319    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="320dcc71-e348-4b3e-9b54-d072549ee9d2" containerName="main"
Sep 10 12:33:15.595409 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.595327    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="f9d781fe-5816-4b44-8819-2b0e77c9fb19" containerName="main"
Sep 10 12:33:15.620605 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.620580    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"]
Sep 10 12:33:15.620773 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.620680    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.626874 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.626854    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"openshift-service-ca.crt\""
Sep 10 12:33:15.627019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.626927    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"default-dockercfg-nlftf\""
Sep 10 12:33:15.627093 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.626866    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"rate-test-model-18c2d2-kserve-self-signed-certs\""
Sep 10 12:33:15.627211 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.627194    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-18c2d2\"/\"kube-root-ca.crt\""
Sep 10 12:33:15.754224 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.754190    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpj2l\" (UniqueName: \"kubernetes.io/projected/8dce3c4c-8214-408d-9149-d096eff5ac7c-kube-api-access-kpj2l\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.754422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.754275    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dce3c4c-8214-408d-9149-d096eff5ac7c-tls-certs\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.754422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.754307    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-dshm\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.754422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.754392    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-home\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.754630 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.754448    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-model-cache\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.754630 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.754475    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-tmp-dir\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855006    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-tmp-dir\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855110    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpj2l\" (UniqueName: \"kubernetes.io/projected/8dce3c4c-8214-408d-9149-d096eff5ac7c-kube-api-access-kpj2l\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855156    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dce3c4c-8214-408d-9149-d096eff5ac7c-tls-certs\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855181    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-dshm\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855212    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-home\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855261 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855242    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-model-cache\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855582 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855422    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-tmp-dir\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855624 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855603    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-model-cache\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.855659 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.855640    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-home\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.857506 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.857486    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-dshm\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.857766 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.857746    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dce3c4c-8214-408d-9149-d096eff5ac7c-tls-certs\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.864491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.864467    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpj2l\" (UniqueName: \"kubernetes.io/projected/8dce3c4c-8214-408d-9149-d096eff5ac7c-kube-api-access-kpj2l\") pod \"rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") " pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:15.931984 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:15.931952    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:16.222972 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:16.220384    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"]
Sep 10 12:33:16.222972 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:33:16.222713    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8dce3c4c_8214_408d_9149_d096eff5ac7c.slice/crio-15138e355f29bf9d1c18d42c39730ec0424eba33754b8999b856ab725eae6455 WatchSource:0}: Error finding container 15138e355f29bf9d1c18d42c39730ec0424eba33754b8999b856ab725eae6455: Status 404 returned error can't find the container with id 15138e355f29bf9d1c18d42c39730ec0424eba33754b8999b856ab725eae6455
Sep 10 12:33:16.878719 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:16.878677    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7" event={"ID":"8dce3c4c-8214-408d-9149-d096eff5ac7c","Type":"ContainerStarted","Data":"70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05"}
Sep 10 12:33:16.879093 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:16.878729    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7" event={"ID":"8dce3c4c-8214-408d-9149-d096eff5ac7c","Type":"ContainerStarted","Data":"15138e355f29bf9d1c18d42c39730ec0424eba33754b8999b856ab725eae6455"}
Sep 10 12:33:16.897443 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:16.897363    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7" podStartSLOduration=1.897342903 podStartE2EDuration="1.897342903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:33:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:33:16.896763218 +0000 UTC m=+1408.093278921" watchObservedRunningTime="2026-09-10 12:33:16.897342903 +0000 UTC m=+1408.093858605"
Sep 10 12:33:21.908233 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:21.908200    2552 generic.go:356] "Generic (PLEG): container finished" podID="972fe1af-6eae-4789-8e38-6bc959916e86" containerID="ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35" exitCode=0
Sep 10 12:33:21.908795 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:21.908278    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" event={"ID":"972fe1af-6eae-4789-8e38-6bc959916e86","Type":"ContainerDied","Data":"ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35"}
Sep 10 12:33:21.908862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:21.908844    2552 scope.go:117] "RemoveContainer" containerID="ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35"
Sep 10 12:33:21.909986 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:21.909957    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:33:21.910071 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:21.910026    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:33:21.910071 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:21.910055    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:33:22.919143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:22.919099    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" event={"ID":"972fe1af-6eae-4789-8e38-6bc959916e86","Type":"ContainerStarted","Data":"72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b"}
Sep 10 12:33:23.564242 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:23.564205    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:23.564416 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:23.564395    2552 patch_prober.go:28] interesting pod/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 12:33:23.564509 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:23.564476    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 12:33:24.564314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:24.564278    2552 patch_prober.go:28] interesting pod/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 12:33:24.564711 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:24.564334    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 12:33:25.563955 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:25.563920    2552 patch_prober.go:28] interesting pod/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 12:33:25.564132 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:25.563986    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 12:33:25.932112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:25.932051    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:25.932112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:25.932113    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:25.945341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:25.945315    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:26.563845 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:26.563807    2552 patch_prober.go:28] interesting pod/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 12:33:26.564017 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:26.563875    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 12:33:26.952509 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:26.952480    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:33:27.563703 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:27.563670    2552 patch_prober.go:28] interesting pod/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused" start-of-body=
Sep 10 12:33:27.563889 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:27.563722    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.78:15021/healthz/ready\": dial tcp 10.134.0.78:15021: connect: connection refused"
Sep 10 12:33:28.567638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:28.567608    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:28.568050 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:28.567878    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:28.568643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:28.568627    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:33:28.949539 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:28.949506    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:33:31.494898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.494861    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"]
Sep 10 12:33:31.499118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.499092    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"
Sep 10 12:33:31.503421 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.503405    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:33:31.503762 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.503743    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:33:31.509090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.509066    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"]
Sep 10 12:33:31.591799 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.591771    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba5c167-aeac-4f6a-9302-d9f2c89c8531-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56\" (UID: \"bba5c167-aeac-4f6a-9302-d9f2c89c8531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"
Sep 10 12:33:31.693198 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.693178    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba5c167-aeac-4f6a-9302-d9f2c89c8531-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56\" (UID: \"bba5c167-aeac-4f6a-9302-d9f2c89c8531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"
Sep 10 12:33:31.693807 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.693784    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba5c167-aeac-4f6a-9302-d9f2c89c8531-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56\" (UID: \"bba5c167-aeac-4f6a-9302-d9f2c89c8531\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"
Sep 10 12:33:31.810635 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:31.810578    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"
Sep 10 12:33:32.081789 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:32.081767    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"]
Sep 10 12:33:32.084083 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:33:32.084057    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbba5c167_aeac_4f6a_9302_d9f2c89c8531.slice/crio-92cfac258e77c5914e45777cba7e0a1d73ee08d7ceb0c004236a8294e8f5590f WatchSource:0}: Error finding container 92cfac258e77c5914e45777cba7e0a1d73ee08d7ceb0c004236a8294e8f5590f: Status 404 returned error can't find the container with id 92cfac258e77c5914e45777cba7e0a1d73ee08d7ceb0c004236a8294e8f5590f
Sep 10 12:33:32.974237 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:32.974207    2552 generic.go:356] "Generic (PLEG): container finished" podID="bba5c167-aeac-4f6a-9302-d9f2c89c8531" containerID="b571b34ee7d0e623114c64fffd1c6c1039625c43e4bfe0f68bbd25b2ff2b3eb2" exitCode=0
Sep 10 12:33:32.974634 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:32.974291    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56" event={"ID":"bba5c167-aeac-4f6a-9302-d9f2c89c8531","Type":"ContainerDied","Data":"b571b34ee7d0e623114c64fffd1c6c1039625c43e4bfe0f68bbd25b2ff2b3eb2"}
Sep 10 12:33:32.974634 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:32.974341    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56" event={"ID":"bba5c167-aeac-4f6a-9302-d9f2c89c8531","Type":"ContainerStarted","Data":"92cfac258e77c5914e45777cba7e0a1d73ee08d7ceb0c004236a8294e8f5590f"}
Sep 10 12:33:34.101810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:34.101788    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"
Sep 10 12:33:34.108934 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:34.108917    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba5c167-aeac-4f6a-9302-d9f2c89c8531-maas-api-service-ca\") pod \"bba5c167-aeac-4f6a-9302-d9f2c89c8531\" (UID: \"bba5c167-aeac-4f6a-9302-d9f2c89c8531\") "
Sep 10 12:33:34.109238 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:34.109217    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bba5c167-aeac-4f6a-9302-d9f2c89c8531-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bba5c167-aeac-4f6a-9302-d9f2c89c8531" (UID: "bba5c167-aeac-4f6a-9302-d9f2c89c8531"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:34.210314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:34.210279    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bba5c167-aeac-4f6a-9302-d9f2c89c8531-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:34.985550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:34.985519    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"
Sep 10 12:33:34.985550 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:34.985535    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56" event={"ID":"bba5c167-aeac-4f6a-9302-d9f2c89c8531","Type":"ContainerDied","Data":"92cfac258e77c5914e45777cba7e0a1d73ee08d7ceb0c004236a8294e8f5590f"}
Sep 10 12:33:34.985917 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:34.985599    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="92cfac258e77c5914e45777cba7e0a1d73ee08d7ceb0c004236a8294e8f5590f"
Sep 10 12:33:47.318111 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.318035    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"]
Sep 10 12:33:47.318492 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.318311    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9" podUID="73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" containerName="payload-pre-processing" containerID="cri-o://0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014" gracePeriod=30
Sep 10 12:33:47.576759 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.576690    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:33:47.590191 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.590166    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"]
Sep 10 12:33:47.591860 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.591836    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-5ff68c-pnm56"]
Sep 10 12:33:47.623443 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.623403    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-plugins-config-volume\") pod \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") "
Sep 10 12:33:47.623558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.623482    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-grrcp\" (UniqueName: \"kubernetes.io/projected/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-kube-api-access-grrcp\") pod \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\" (UID: \"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6\") "
Sep 10 12:33:47.623849 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.623825    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" (UID: "73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:47.625802 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.625777    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-kube-api-access-grrcp" (OuterVolumeSpecName: "kube-api-access-grrcp") pod "73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" (UID: "73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6"). InnerVolumeSpecName "kube-api-access-grrcp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:33:47.724369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.724337    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:47.724369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:47.724369    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grrcp\" (UniqueName: \"kubernetes.io/projected/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6-kube-api-access-grrcp\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:48.051774 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.051744    2552 generic.go:356] "Generic (PLEG): container finished" podID="73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" containerID="0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014" exitCode=0
Sep 10 12:33:48.051945 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.051793    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"
Sep 10 12:33:48.051945 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.051836    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9" event={"ID":"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6","Type":"ContainerDied","Data":"0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014"}
Sep 10 12:33:48.051945 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.051897    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9" event={"ID":"73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6","Type":"ContainerDied","Data":"49bc3ce0b3f3d49e03d38eff34e2e3f8d6ba9e557d20224f13d343307b298c24"}
Sep 10 12:33:48.051945 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.051925    2552 scope.go:117] "RemoveContainer" containerID="0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014"
Sep 10 12:33:48.060811 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.060794    2552 scope.go:117] "RemoveContainer" containerID="0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014"
Sep 10 12:33:48.061069 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:33:48.061049    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014\": container with ID starting with 0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014 not found: ID does not exist" containerID="0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014"
Sep 10 12:33:48.061127 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.061081    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014"} err="failed to get container status \"0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014\": rpc error: code = NotFound desc = could not find container \"0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014\": container with ID starting with 0197055a976946ed636cb63f012130a1a30f93568721c3c33423788048d72014 not found: ID does not exist"
Sep 10 12:33:48.081554 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.081529    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"]
Sep 10 12:33:48.083150 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.083127    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-5ff68c-9d76c4658-drpg9"]
Sep 10 12:33:48.380789 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.380753    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"]
Sep 10 12:33:48.381331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.381311    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" containerName="payload-pre-processing"
Sep 10 12:33:48.381419 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.381334    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" containerName="payload-pre-processing"
Sep 10 12:33:48.381419 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.381347    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bba5c167-aeac-4f6a-9302-d9f2c89c8531" containerName="revoke-keys"
Sep 10 12:33:48.381419 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.381355    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bba5c167-aeac-4f6a-9302-d9f2c89c8531" containerName="revoke-keys"
Sep 10 12:33:48.381612 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.381468    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" containerName="payload-pre-processing"
Sep 10 12:33:48.381612 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.381485    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bba5c167-aeac-4f6a-9302-d9f2c89c8531" containerName="revoke-keys"
Sep 10 12:33:48.384488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.384469    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.388027 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.388006    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"rate-test-model-f89a79-kserve-self-signed-certs\""
Sep 10 12:33:48.388127 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.388027    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"kube-root-ca.crt\""
Sep 10 12:33:48.388798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.388780    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"default-dockercfg-rfdfk\""
Sep 10 12:33:48.388798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.388793    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-f89a79\"/\"openshift-service-ca.crt\""
Sep 10 12:33:48.404126 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.404107    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"]
Sep 10 12:33:48.431107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.431083    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070e50f4-8668-449d-af63-8dfe4436d432-tls-certs\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.431240 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.431112    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dgpk\" (UniqueName: \"kubernetes.io/projected/070e50f4-8668-449d-af63-8dfe4436d432-kube-api-access-8dgpk\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.431240 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.431148    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-dshm\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.431374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.431243    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-model-cache\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.431374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.431301    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-home\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.431374 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.431364    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-tmp-dir\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532371 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532347    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-dshm\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532484 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532385    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-model-cache\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532484 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532409    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-home\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532563 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532525    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-tmp-dir\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532563 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532555    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070e50f4-8668-449d-af63-8dfe4436d432-tls-certs\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532659 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532577    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dgpk\" (UniqueName: \"kubernetes.io/projected/070e50f4-8668-449d-af63-8dfe4436d432-kube-api-access-8dgpk\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532819 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532784    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-model-cache\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532843    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-home\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.532931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.532900    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-tmp-dir\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.534699 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.534681    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-dshm\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.535287 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.535265    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070e50f4-8668-449d-af63-8dfe4436d432-tls-certs\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.542069 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.542040    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dgpk\" (UniqueName: \"kubernetes.io/projected/070e50f4-8668-449d-af63-8dfe4436d432-kube-api-access-8dgpk\") pod \"rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") " pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.693714 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.693647    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:48.972629 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:48.972607    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"]
Sep 10 12:33:48.975097 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:33:48.975070    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod070e50f4_8668_449d_af63_8dfe4436d432.slice/crio-57b7e4efaec2314a6cedaabe3c6ba3d624d0e8dd8660de95549287a4eb3c8b7d WatchSource:0}: Error finding container 57b7e4efaec2314a6cedaabe3c6ba3d624d0e8dd8660de95549287a4eb3c8b7d: Status 404 returned error can't find the container with id 57b7e4efaec2314a6cedaabe3c6ba3d624d0e8dd8660de95549287a4eb3c8b7d
Sep 10 12:33:49.060258 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:49.059910    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48" event={"ID":"070e50f4-8668-449d-af63-8dfe4436d432","Type":"ContainerStarted","Data":"d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a"}
Sep 10 12:33:49.060258 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:49.059956    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48" event={"ID":"070e50f4-8668-449d-af63-8dfe4436d432","Type":"ContainerStarted","Data":"57b7e4efaec2314a6cedaabe3c6ba3d624d0e8dd8660de95549287a4eb3c8b7d"}
Sep 10 12:33:49.083500 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:49.083459    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48" podStartSLOduration=1.083422607 podStartE2EDuration="1.083422607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:33:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:33:49.082183531 +0000 UTC m=+1440.278699230" watchObservedRunningTime="2026-09-10 12:33:49.083422607 +0000 UTC m=+1440.279938306"
Sep 10 12:33:49.245024 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:49.244951    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6" path="/var/lib/kubelet/pods/73aefad6-a5ee-4051-87d0-7d6e8e0dc8d6/volumes"
Sep 10 12:33:49.245286 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:49.245273    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bba5c167-aeac-4f6a-9302-d9f2c89c8531" path="/var/lib/kubelet/pods/bba5c167-aeac-4f6a-9302-d9f2c89c8531/volumes"
Sep 10 12:33:49.466563 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:49.466527    2552 scope.go:117] "RemoveContainer" containerID="69227f7b72d74341fd99c45328b571b9e3cb2cd3aefe02dc5e19279fd02b9dbc"
Sep 10 12:33:49.475291 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:49.475274    2552 scope.go:117] "RemoveContainer" containerID="5b0ad20320394c3698cd673eb49b8419a0c7f53c98655bf4b3c21a0cb3c58de0"
Sep 10 12:33:52.493518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:52.493485    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:33:55.393626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.393591    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"]
Sep 10 12:33:55.398959 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.398942    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"
Sep 10 12:33:55.401843 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.401819    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:33:55.401972 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.401951    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:33:55.406480 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.406457    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"]
Sep 10 12:33:55.494748 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.494719    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb411e4-b75e-486b-a266-ee9fd3840bf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x\" (UID: \"bdb411e4-b75e-486b-a266-ee9fd3840bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"
Sep 10 12:33:55.595650 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.595620    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb411e4-b75e-486b-a266-ee9fd3840bf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x\" (UID: \"bdb411e4-b75e-486b-a266-ee9fd3840bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"
Sep 10 12:33:55.596193 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.596171    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb411e4-b75e-486b-a266-ee9fd3840bf6-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x\" (UID: \"bdb411e4-b75e-486b-a266-ee9fd3840bf6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"
Sep 10 12:33:55.711449 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.711370    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"
Sep 10 12:33:55.981978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:55.981957    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"]
Sep 10 12:33:55.984152 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:33:55.984122    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbdb411e4_b75e_486b_a266_ee9fd3840bf6.slice/crio-556e64c3976aec9d5afb506099fa9c085dda036551ac3f637ca30dc08239535a WatchSource:0}: Error finding container 556e64c3976aec9d5afb506099fa9c085dda036551ac3f637ca30dc08239535a: Status 404 returned error can't find the container with id 556e64c3976aec9d5afb506099fa9c085dda036551ac3f637ca30dc08239535a
Sep 10 12:33:56.098473 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:56.098412    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x" event={"ID":"bdb411e4-b75e-486b-a266-ee9fd3840bf6","Type":"ContainerStarted","Data":"556e64c3976aec9d5afb506099fa9c085dda036551ac3f637ca30dc08239535a"}
Sep 10 12:33:57.105558 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:57.105483    2552 generic.go:356] "Generic (PLEG): container finished" podID="bdb411e4-b75e-486b-a266-ee9fd3840bf6" containerID="79775e4bb390334bb282f90f3a8d8cefc140fe91519b55e84ca1301104b1bcf9" exitCode=0
Sep 10 12:33:57.105915 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:57.105574    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x" event={"ID":"bdb411e4-b75e-486b-a266-ee9fd3840bf6","Type":"ContainerDied","Data":"79775e4bb390334bb282f90f3a8d8cefc140fe91519b55e84ca1301104b1bcf9"}
Sep 10 12:33:58.242882 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:58.242858    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"
Sep 10 12:33:58.327299 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:58.327272    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb411e4-b75e-486b-a266-ee9fd3840bf6-maas-api-service-ca\") pod \"bdb411e4-b75e-486b-a266-ee9fd3840bf6\" (UID: \"bdb411e4-b75e-486b-a266-ee9fd3840bf6\") "
Sep 10 12:33:58.327615 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:58.327593    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bdb411e4-b75e-486b-a266-ee9fd3840bf6-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "bdb411e4-b75e-486b-a266-ee9fd3840bf6" (UID: "bdb411e4-b75e-486b-a266-ee9fd3840bf6"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:33:58.328352 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:58.328318    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/bdb411e4-b75e-486b-a266-ee9fd3840bf6-maas-api-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:33:58.694029 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:58.694000    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:58.694207 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:58.694054    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:58.710999 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:58.710975    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:33:59.117361 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:59.117337    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"
Sep 10 12:33:59.117557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:59.117384    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x" event={"ID":"bdb411e4-b75e-486b-a266-ee9fd3840bf6","Type":"ContainerDied","Data":"556e64c3976aec9d5afb506099fa9c085dda036551ac3f637ca30dc08239535a"}
Sep 10 12:33:59.117557 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:59.117453    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="556e64c3976aec9d5afb506099fa9c085dda036551ac3f637ca30dc08239535a"
Sep 10 12:33:59.129214 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:33:59.129197    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:34:11.605372 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:11.605336    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"]
Sep 10 12:34:11.608339 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:11.608312    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-da7b8f-gzr7x"]
Sep 10 12:34:13.241912 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:13.241868    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bdb411e4-b75e-486b-a266-ee9fd3840bf6" path="/var/lib/kubelet/pods/bdb411e4-b75e-486b-a266-ee9fd3840bf6/volumes"
Sep 10 12:34:16.553728 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:16.553695    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:34:19.292348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:19.292316    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"]
Sep 10 12:34:19.292806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:19.292570    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy" containerID="cri-o://72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b" gracePeriod=30
Sep 10 12:34:24.540352 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.540326    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:34:24.641987 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.641965    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972fe1af-6eae-4789-8e38-6bc959916e86-istiod-ca-cert\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642118 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642044    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-envoy\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642168 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642145    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-certs\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642230 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642185    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5jm8z\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-kube-api-access-5jm8z\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642289 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642260    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-istio-token\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642362 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642293    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972fe1af-6eae-4789-8e38-6bc959916e86-istio-podinfo\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642362 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642326    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-data\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642371    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-socket\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642413    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:24.642510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642455    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-credential-socket\") pod \"972fe1af-6eae-4789-8e38-6bc959916e86\" (UID: \"972fe1af-6eae-4789-8e38-6bc959916e86\") "
Sep 10 12:34:24.642510 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642485    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/972fe1af-6eae-4789-8e38-6bc959916e86-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:34:24.642725 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642697    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-data" (OuterVolumeSpecName: "istio-data") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:24.642901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642867    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.642901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642897    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.643098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642910    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/972fe1af-6eae-4789-8e38-6bc959916e86-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.643098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642918    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:24.643098 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.642920    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:24.644410 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.644381    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:24.644532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.644440    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-kube-api-access-5jm8z" (OuterVolumeSpecName: "kube-api-access-5jm8z") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "kube-api-access-5jm8z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:34:24.644532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.644506    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/972fe1af-6eae-4789-8e38-6bc959916e86-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:34:24.644680 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.644660    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-istio-token" (OuterVolumeSpecName: "istio-token") pod "972fe1af-6eae-4789-8e38-6bc959916e86" (UID: "972fe1af-6eae-4789-8e38-6bc959916e86"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:34:24.743757 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.743735    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.743757 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.743755    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/972fe1af-6eae-4789-8e38-6bc959916e86-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.743901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.743764    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.743901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.743772    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.743901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.743782    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/972fe1af-6eae-4789-8e38-6bc959916e86-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:24.743901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:24.743791    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5jm8z\" (UniqueName: \"kubernetes.io/projected/972fe1af-6eae-4789-8e38-6bc959916e86-kube-api-access-5jm8z\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:25.256729 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.256706    2552 generic.go:356] "Generic (PLEG): container finished" podID="972fe1af-6eae-4789-8e38-6bc959916e86" containerID="72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b" exitCode=0
Sep 10 12:34:25.256854 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.256753    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"
Sep 10 12:34:25.256854 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.256757    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" event={"ID":"972fe1af-6eae-4789-8e38-6bc959916e86","Type":"ContainerDied","Data":"72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b"}
Sep 10 12:34:25.256931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.256887    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4" event={"ID":"972fe1af-6eae-4789-8e38-6bc959916e86","Type":"ContainerDied","Data":"c0cccf13a0c7cebe4ebd815df77419dafbdb41215f58712943b6d6f1b0412be5"}
Sep 10 12:34:25.256931 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.256926    2552 scope.go:117] "RemoveContainer" containerID="72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b"
Sep 10 12:34:25.265895 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.265875    2552 scope.go:117] "RemoveContainer" containerID="ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35"
Sep 10 12:34:25.274198 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.274181    2552 scope.go:117] "RemoveContainer" containerID="72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b"
Sep 10 12:34:25.274528 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:34:25.274495    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b\": container with ID starting with 72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b not found: ID does not exist" containerID="72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b"
Sep 10 12:34:25.274623 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.274532    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b"} err="failed to get container status \"72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b\": rpc error: code = NotFound desc = could not find container \"72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b\": container with ID starting with 72f3d7eeb26f33da9050a3ab578ebf8634b2cc5fd269d09b046f7b72f167876b not found: ID does not exist"
Sep 10 12:34:25.274623 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.274555    2552 scope.go:117] "RemoveContainer" containerID="ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35"
Sep 10 12:34:25.274884 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:34:25.274861    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35\": container with ID starting with ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35 not found: ID does not exist" containerID="ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35"
Sep 10 12:34:25.274963 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.274889    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35"} err="failed to get container status \"ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35\": rpc error: code = NotFound desc = could not find container \"ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35\": container with ID starting with ce58df8a60c92757a155eafe6ad30be1fd8a4fc9e0013da24e02d45f99218f35 not found: ID does not exist"
Sep 10 12:34:25.283877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.283852    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"]
Sep 10 12:34:25.290006 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:25.289986    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-da7b8f-openshift-default-5dbffd594-t82n4"]
Sep 10 12:34:27.247762 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:27.247721    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" path="/var/lib/kubelet/pods/972fe1af-6eae-4789-8e38-6bc959916e86/volumes"
Sep 10 12:34:30.559836 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:30.559806    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"]
Sep 10 12:34:30.560321 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:30.560003    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64" podUID="bbd6433c-735b-439f-b53b-8be16b6746c6" containerName="payload-pre-processing" containerID="cri-o://e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610" gracePeriod=30
Sep 10 12:34:30.812682 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:30.812628    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:34:30.904600 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:30.904568    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ss6xr\" (UniqueName: \"kubernetes.io/projected/bbd6433c-735b-439f-b53b-8be16b6746c6-kube-api-access-ss6xr\") pod \"bbd6433c-735b-439f-b53b-8be16b6746c6\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") "
Sep 10 12:34:30.904755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:30.904621    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd6433c-735b-439f-b53b-8be16b6746c6-plugins-config-volume\") pod \"bbd6433c-735b-439f-b53b-8be16b6746c6\" (UID: \"bbd6433c-735b-439f-b53b-8be16b6746c6\") "
Sep 10 12:34:30.904994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:30.904972    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bbd6433c-735b-439f-b53b-8be16b6746c6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bbd6433c-735b-439f-b53b-8be16b6746c6" (UID: "bbd6433c-735b-439f-b53b-8be16b6746c6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:34:30.906766 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:30.906737    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bbd6433c-735b-439f-b53b-8be16b6746c6-kube-api-access-ss6xr" (OuterVolumeSpecName: "kube-api-access-ss6xr") pod "bbd6433c-735b-439f-b53b-8be16b6746c6" (UID: "bbd6433c-735b-439f-b53b-8be16b6746c6"). InnerVolumeSpecName "kube-api-access-ss6xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:34:31.005338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.005311    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ss6xr\" (UniqueName: \"kubernetes.io/projected/bbd6433c-735b-439f-b53b-8be16b6746c6-kube-api-access-ss6xr\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:31.005338 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.005332    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bbd6433c-735b-439f-b53b-8be16b6746c6-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:31.289787 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.289759    2552 generic.go:356] "Generic (PLEG): container finished" podID="bbd6433c-735b-439f-b53b-8be16b6746c6" containerID="e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610" exitCode=0
Sep 10 12:34:31.289914 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.289797    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"
Sep 10 12:34:31.289914 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.289841    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64" event={"ID":"bbd6433c-735b-439f-b53b-8be16b6746c6","Type":"ContainerDied","Data":"e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610"}
Sep 10 12:34:31.289914 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.289892    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64" event={"ID":"bbd6433c-735b-439f-b53b-8be16b6746c6","Type":"ContainerDied","Data":"7bafd456c2eede029d87f41dab106b2ecde61b8bd78f33d7243d1342b4c3bf52"}
Sep 10 12:34:31.290027 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.289914    2552 scope.go:117] "RemoveContainer" containerID="e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610"
Sep 10 12:34:31.298552 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.298519    2552 scope.go:117] "RemoveContainer" containerID="e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610"
Sep 10 12:34:31.298873 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:34:31.298841    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610\": container with ID starting with e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610 not found: ID does not exist" containerID="e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610"
Sep 10 12:34:31.298946 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.298879    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610"} err="failed to get container status \"e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610\": rpc error: code = NotFound desc = could not find container \"e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610\": container with ID starting with e16f83855b82b7aa2dee042dfd86a34d4cd994b04bdf6391372f1d4b71ffc610 not found: ID does not exist"
Sep 10 12:34:31.309072 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.309049    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"]
Sep 10 12:34:31.312526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:31.312506    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-d1e553-796d4767d6-tzw64"]
Sep 10 12:34:33.242388 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:33.242353    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bbd6433c-735b-439f-b53b-8be16b6746c6" path="/var/lib/kubelet/pods/bbd6433c-735b-439f-b53b-8be16b6746c6/volumes"
Sep 10 12:34:34.056547 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:34.056514    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:34:38.768695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:38.768660    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"]
Sep 10 12:34:39.662825 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:39.662796    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"]
Sep 10 12:34:49.166792 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:49.166758    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:34:49.843518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:49.843442    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"]
Sep 10 12:34:49.851643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:49.851622    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-a08ea2-29817390-s5m8l"]
Sep 10 12:34:49.917409 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:49.917384    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"]
Sep 10 12:34:49.917656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:49.917612    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5" podUID="3be77961-a2fb-4439-b06e-d27ecf3ff8b0" containerName="payload-pre-processing" containerID="cri-o://e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4" gracePeriod=30
Sep 10 12:34:50.173570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.173546    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:34:50.267090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.267059    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-plugins-config-volume\") pod \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") "
Sep 10 12:34:50.267248 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.267098    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szmjk\" (UniqueName: \"kubernetes.io/projected/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-kube-api-access-szmjk\") pod \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\" (UID: \"3be77961-a2fb-4439-b06e-d27ecf3ff8b0\") "
Sep 10 12:34:50.267526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.267497    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3be77961-a2fb-4439-b06e-d27ecf3ff8b0" (UID: "3be77961-a2fb-4439-b06e-d27ecf3ff8b0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:34:50.269378 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.269361    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-kube-api-access-szmjk" (OuterVolumeSpecName: "kube-api-access-szmjk") pod "3be77961-a2fb-4439-b06e-d27ecf3ff8b0" (UID: "3be77961-a2fb-4439-b06e-d27ecf3ff8b0"). InnerVolumeSpecName "kube-api-access-szmjk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:34:50.368730 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.368700    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:50.368730 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.368728    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szmjk\" (UniqueName: \"kubernetes.io/projected/3be77961-a2fb-4439-b06e-d27ecf3ff8b0-kube-api-access-szmjk\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:50.386405 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.386379    2552 generic.go:356] "Generic (PLEG): container finished" podID="3be77961-a2fb-4439-b06e-d27ecf3ff8b0" containerID="e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4" exitCode=0
Sep 10 12:34:50.386521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.386421    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"
Sep 10 12:34:50.386521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.386470    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5" event={"ID":"3be77961-a2fb-4439-b06e-d27ecf3ff8b0","Type":"ContainerDied","Data":"e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4"}
Sep 10 12:34:50.386521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.386518    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5" event={"ID":"3be77961-a2fb-4439-b06e-d27ecf3ff8b0","Type":"ContainerDied","Data":"1dcb5f71421b09414d552d32183e5ba34ddc834875cb743b3d1f457397a05a84"}
Sep 10 12:34:50.386655 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.386543    2552 scope.go:117] "RemoveContainer" containerID="e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4"
Sep 10 12:34:50.395925 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.395908    2552 scope.go:117] "RemoveContainer" containerID="e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4"
Sep 10 12:34:50.396203 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:34:50.396176    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4\": container with ID starting with e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4 not found: ID does not exist" containerID="e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4"
Sep 10 12:34:50.396258 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.396201    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4"} err="failed to get container status \"e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4\": rpc error: code = NotFound desc = could not find container \"e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4\": container with ID starting with e4c347ef8494ac510123a3c3057902e0f5e83cc02ed8b4128c8a6c2184e349a4 not found: ID does not exist"
Sep 10 12:34:50.436852 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.436796    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"]
Sep 10 12:34:50.443266 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:50.443246    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-a08ea2-84f8b84c4b-nxfz5"]
Sep 10 12:34:51.242140 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:51.242113    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00bf1aa6-faf2-49a8-ab42-75e107726345" path="/var/lib/kubelet/pods/00bf1aa6-faf2-49a8-ab42-75e107726345/volumes"
Sep 10 12:34:51.242504 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:51.242416    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3be77961-a2fb-4439-b06e-d27ecf3ff8b0" path="/var/lib/kubelet/pods/3be77961-a2fb-4439-b06e-d27ecf3ff8b0/volumes"
Sep 10 12:34:53.787737 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:53.787679    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7" podUID="8dce3c4c-8214-408d-9149-d096eff5ac7c" containerName="main" containerID="cri-o://70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05" gracePeriod=45
Sep 10 12:34:54.041247 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.041195    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:34:54.200870 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.200845    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dce3c4c-8214-408d-9149-d096eff5ac7c-tls-certs\") pod \"8dce3c4c-8214-408d-9149-d096eff5ac7c\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") "
Sep 10 12:34:54.201015 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.200921    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-home\") pod \"8dce3c4c-8214-408d-9149-d096eff5ac7c\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") "
Sep 10 12:34:54.201015 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.200951    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-tmp-dir\") pod \"8dce3c4c-8214-408d-9149-d096eff5ac7c\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") "
Sep 10 12:34:54.201015 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.200993    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-model-cache\") pod \"8dce3c4c-8214-408d-9149-d096eff5ac7c\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") "
Sep 10 12:34:54.201184 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201018    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-dshm\") pod \"8dce3c4c-8214-408d-9149-d096eff5ac7c\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") "
Sep 10 12:34:54.201184 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201057    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kpj2l\" (UniqueName: \"kubernetes.io/projected/8dce3c4c-8214-408d-9149-d096eff5ac7c-kube-api-access-kpj2l\") pod \"8dce3c4c-8214-408d-9149-d096eff5ac7c\" (UID: \"8dce3c4c-8214-408d-9149-d096eff5ac7c\") "
Sep 10 12:34:54.201288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201224    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-home" (OuterVolumeSpecName: "home") pod "8dce3c4c-8214-408d-9149-d096eff5ac7c" (UID: "8dce3c4c-8214-408d-9149-d096eff5ac7c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:54.201288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201226    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8dce3c4c-8214-408d-9149-d096eff5ac7c" (UID: "8dce3c4c-8214-408d-9149-d096eff5ac7c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:54.201288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201249    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-model-cache" (OuterVolumeSpecName: "model-cache") pod "8dce3c4c-8214-408d-9149-d096eff5ac7c" (UID: "8dce3c4c-8214-408d-9149-d096eff5ac7c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:54.201423 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201358    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:54.201423 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201375    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:54.201423 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.201389    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:54.203090 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.203066    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8dce3c4c-8214-408d-9149-d096eff5ac7c-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8dce3c4c-8214-408d-9149-d096eff5ac7c" (UID: "8dce3c4c-8214-408d-9149-d096eff5ac7c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:34:54.203203 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.203163    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-dshm" (OuterVolumeSpecName: "dshm") pod "8dce3c4c-8214-408d-9149-d096eff5ac7c" (UID: "8dce3c4c-8214-408d-9149-d096eff5ac7c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:54.203303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.203274    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8dce3c4c-8214-408d-9149-d096eff5ac7c-kube-api-access-kpj2l" (OuterVolumeSpecName: "kube-api-access-kpj2l") pod "8dce3c4c-8214-408d-9149-d096eff5ac7c" (UID: "8dce3c4c-8214-408d-9149-d096eff5ac7c"). InnerVolumeSpecName "kube-api-access-kpj2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:34:54.302498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.302445    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpj2l\" (UniqueName: \"kubernetes.io/projected/8dce3c4c-8214-408d-9149-d096eff5ac7c-kube-api-access-kpj2l\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:54.302498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.302469    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8dce3c4c-8214-408d-9149-d096eff5ac7c-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:54.302498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.302480    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8dce3c4c-8214-408d-9149-d096eff5ac7c-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:54.411821 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.411792    2552 generic.go:356] "Generic (PLEG): container finished" podID="8dce3c4c-8214-408d-9149-d096eff5ac7c" containerID="70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05" exitCode=0
Sep 10 12:34:54.411912 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.411836    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"
Sep 10 12:34:54.411912 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.411880    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7" event={"ID":"8dce3c4c-8214-408d-9149-d096eff5ac7c","Type":"ContainerDied","Data":"70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05"}
Sep 10 12:34:54.411992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.411916    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7" event={"ID":"8dce3c4c-8214-408d-9149-d096eff5ac7c","Type":"ContainerDied","Data":"15138e355f29bf9d1c18d42c39730ec0424eba33754b8999b856ab725eae6455"}
Sep 10 12:34:54.411992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.411935    2552 scope.go:117] "RemoveContainer" containerID="70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05"
Sep 10 12:34:54.420638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.420623    2552 scope.go:117] "RemoveContainer" containerID="70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05"
Sep 10 12:34:54.420887 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:34:54.420866    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05\": container with ID starting with 70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05 not found: ID does not exist" containerID="70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05"
Sep 10 12:34:54.420941 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.420897    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05"} err="failed to get container status \"70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05\": rpc error: code = NotFound desc = could not find container \"70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05\": container with ID starting with 70d9e6f3de47e612bc2de2251f83813bc68b3ec4eeea8910be3ae73eb6ad5b05 not found: ID does not exist"
Sep 10 12:34:54.441873 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.441851    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"]
Sep 10 12:34:54.443561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.443544    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-18c2d2/rate-test-model-18c2d2-kserve-7ccc57c8c5-wjtm7"]
Sep 10 12:34:54.682208 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.682175    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48" podUID="070e50f4-8668-449d-af63-8dfe4436d432" containerName="main" containerID="cri-o://d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a" gracePeriod=45
Sep 10 12:34:54.926014 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:54.925988    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:34:55.008095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008022    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-model-cache\") pod \"070e50f4-8668-449d-af63-8dfe4436d432\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") "
Sep 10 12:34:55.008095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008065    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070e50f4-8668-449d-af63-8dfe4436d432-tls-certs\") pod \"070e50f4-8668-449d-af63-8dfe4436d432\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") "
Sep 10 12:34:55.008297 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008106    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8dgpk\" (UniqueName: \"kubernetes.io/projected/070e50f4-8668-449d-af63-8dfe4436d432-kube-api-access-8dgpk\") pod \"070e50f4-8668-449d-af63-8dfe4436d432\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") "
Sep 10 12:34:55.008297 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008173    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-dshm\") pod \"070e50f4-8668-449d-af63-8dfe4436d432\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") "
Sep 10 12:34:55.008297 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008213    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-home\") pod \"070e50f4-8668-449d-af63-8dfe4436d432\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") "
Sep 10 12:34:55.008297 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008236    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-tmp-dir\") pod \"070e50f4-8668-449d-af63-8dfe4436d432\" (UID: \"070e50f4-8668-449d-af63-8dfe4436d432\") "
Sep 10 12:34:55.008555 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008290    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-model-cache" (OuterVolumeSpecName: "model-cache") pod "070e50f4-8668-449d-af63-8dfe4436d432" (UID: "070e50f4-8668-449d-af63-8dfe4436d432"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:55.008555 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008538    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "070e50f4-8668-449d-af63-8dfe4436d432" (UID: "070e50f4-8668-449d-af63-8dfe4436d432"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:55.008658 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008590    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:55.008771 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.008747    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-home" (OuterVolumeSpecName: "home") pod "070e50f4-8668-449d-af63-8dfe4436d432" (UID: "070e50f4-8668-449d-af63-8dfe4436d432"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:55.010239 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.010207    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/070e50f4-8668-449d-af63-8dfe4436d432-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "070e50f4-8668-449d-af63-8dfe4436d432" (UID: "070e50f4-8668-449d-af63-8dfe4436d432"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:34:55.010405 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.010384    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-dshm" (OuterVolumeSpecName: "dshm") pod "070e50f4-8668-449d-af63-8dfe4436d432" (UID: "070e50f4-8668-449d-af63-8dfe4436d432"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:34:55.010405 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.010396    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/070e50f4-8668-449d-af63-8dfe4436d432-kube-api-access-8dgpk" (OuterVolumeSpecName: "kube-api-access-8dgpk") pod "070e50f4-8668-449d-af63-8dfe4436d432" (UID: "070e50f4-8668-449d-af63-8dfe4436d432"). InnerVolumeSpecName "kube-api-access-8dgpk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:34:55.109148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.109120    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:55.109148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.109143    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:55.109148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.109153    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/070e50f4-8668-449d-af63-8dfe4436d432-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:55.109341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.109162    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/070e50f4-8668-449d-af63-8dfe4436d432-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:55.109341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.109171    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dgpk\" (UniqueName: \"kubernetes.io/projected/070e50f4-8668-449d-af63-8dfe4436d432-kube-api-access-8dgpk\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:34:55.243252 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.243224    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8dce3c4c-8214-408d-9149-d096eff5ac7c" path="/var/lib/kubelet/pods/8dce3c4c-8214-408d-9149-d096eff5ac7c/volumes"
Sep 10 12:34:55.424518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.424488    2552 generic.go:356] "Generic (PLEG): container finished" podID="070e50f4-8668-449d-af63-8dfe4436d432" containerID="d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a" exitCode=0
Sep 10 12:34:55.424664 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.424544    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"
Sep 10 12:34:55.424664 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.424576    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48" event={"ID":"070e50f4-8668-449d-af63-8dfe4436d432","Type":"ContainerDied","Data":"d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a"}
Sep 10 12:34:55.424664 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.424624    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48" event={"ID":"070e50f4-8668-449d-af63-8dfe4436d432","Type":"ContainerDied","Data":"57b7e4efaec2314a6cedaabe3c6ba3d624d0e8dd8660de95549287a4eb3c8b7d"}
Sep 10 12:34:55.424664 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.424646    2552 scope.go:117] "RemoveContainer" containerID="d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a"
Sep 10 12:34:55.433263 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.433247    2552 scope.go:117] "RemoveContainer" containerID="d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a"
Sep 10 12:34:55.433585 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:34:55.433563    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a\": container with ID starting with d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a not found: ID does not exist" containerID="d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a"
Sep 10 12:34:55.433694 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.433589    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a"} err="failed to get container status \"d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a\": rpc error: code = NotFound desc = could not find container \"d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a\": container with ID starting with d0bfd50a08e1086a5991b11d799a4aa0c6069c4d03dd8f72678942128af89a6a not found: ID does not exist"
Sep 10 12:34:55.443256 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.443235    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"]
Sep 10 12:34:55.449586 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:55.449567    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-f89a79/rate-test-model-f89a79-kserve-84fbdc5cb7-vjq48"]
Sep 10 12:34:57.243395 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:57.243366    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="070e50f4-8668-449d-af63-8dfe4436d432" path="/var/lib/kubelet/pods/070e50f4-8668-449d-af63-8dfe4436d432/volumes"
Sep 10 12:34:57.326281 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:57.326248    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"]
Sep 10 12:34:57.326503 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:57.326469    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy" containerID="cri-o://e0261e62000ddbeb3da46d68533e283db7c51b296b89806ecd1fc7b34ff5d761" gracePeriod=30
Sep 10 12:34:58.073091 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:34:58.073059    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:35:02.462544 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.462518    2552 generic.go:356] "Generic (PLEG): container finished" podID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerID="e0261e62000ddbeb3da46d68533e283db7c51b296b89806ecd1fc7b34ff5d761" exitCode=0
Sep 10 12:35:02.462899 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.462547    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" event={"ID":"18e963c5-8aea-42d7-bcb5-71be9e9331b0","Type":"ContainerDied","Data":"e0261e62000ddbeb3da46d68533e283db7c51b296b89806ecd1fc7b34ff5d761"}
Sep 10 12:35:02.462899 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.462620    2552 scope.go:117] "RemoveContainer" containerID="7ad6408505c6c66cc7836f521b1b38bb5a4fcde4eaf1586b518b0dd1b552f819"
Sep 10 12:35:02.583269 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.583246    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:35:02.668883 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.668851    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-certs\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.668883 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.668881    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istiod-ca-cert\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669071 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.668920    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rjzz9\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-kube-api-access-rjzz9\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669071 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669042    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-podinfo\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669084    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-credential-socket\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669108    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-data\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669180 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669146    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-envoy\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669186    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-socket\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669191    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:35:02.669328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669234    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:35:02.669328 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669305    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-token\") pod \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\" (UID: \"18e963c5-8aea-42d7-bcb5-71be9e9331b0\") "
Sep 10 12:35:02.669577 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669342    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:35:02.669577 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669412    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:35:02.669577 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669461    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-data" (OuterVolumeSpecName: "istio-data") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:35:02.669898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669669    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.669898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669690    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.669898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669704    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.669898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669715    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.669898 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.669728    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.671351 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.671320    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-kube-api-access-rjzz9" (OuterVolumeSpecName: "kube-api-access-rjzz9") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "kube-api-access-rjzz9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:35:02.671529 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.671350    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:35:02.671529 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.671410    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:35:02.671617 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.671535    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-token" (OuterVolumeSpecName: "istio-token") pod "18e963c5-8aea-42d7-bcb5-71be9e9331b0" (UID: "18e963c5-8aea-42d7-bcb5-71be9e9331b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:35:02.770900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.770870    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.770900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.770896    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.771074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.770906    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rjzz9\" (UniqueName: \"kubernetes.io/projected/18e963c5-8aea-42d7-bcb5-71be9e9331b0-kube-api-access-rjzz9\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:02.771074 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:02.770916    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/18e963c5-8aea-42d7-bcb5-71be9e9331b0-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:35:03.469355 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:03.469317    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"
Sep 10 12:35:03.469803 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:03.469353    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv" event={"ID":"18e963c5-8aea-42d7-bcb5-71be9e9331b0","Type":"ContainerDied","Data":"d076e8ef02b505752381a25dd162f93be052a41207c96f307496ba094615de64"}
Sep 10 12:35:03.469803 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:03.469411    2552 scope.go:117] "RemoveContainer" containerID="e0261e62000ddbeb3da46d68533e283db7c51b296b89806ecd1fc7b34ff5d761"
Sep 10 12:35:03.493667 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:03.493615    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"]
Sep 10 12:35:03.495656 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:03.495636    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-a08ea2-openshift-default-7cb77f7dc4-vjnnv"]
Sep 10 12:35:05.241879 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:05.241842    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" path="/var/lib/kubelet/pods/18e963c5-8aea-42d7-bcb5-71be9e9331b0/volumes"
Sep 10 12:35:09.316390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316356    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"]
Sep 10 12:35:09.316754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316716    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy"
Sep 10 12:35:09.316754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316729    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy"
Sep 10 12:35:09.316754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316737    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy"
Sep 10 12:35:09.316754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316742    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy"
Sep 10 12:35:09.316754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316750    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="070e50f4-8668-449d-af63-8dfe4436d432" containerName="main"
Sep 10 12:35:09.316754 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316755    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="070e50f4-8668-449d-af63-8dfe4436d432" containerName="main"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316762    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3be77961-a2fb-4439-b06e-d27ecf3ff8b0" containerName="payload-pre-processing"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316768    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="3be77961-a2fb-4439-b06e-d27ecf3ff8b0" containerName="payload-pre-processing"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316775    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bbd6433c-735b-439f-b53b-8be16b6746c6" containerName="payload-pre-processing"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316782    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bbd6433c-735b-439f-b53b-8be16b6746c6" containerName="payload-pre-processing"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316792    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316799    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316806    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bdb411e4-b75e-486b-a266-ee9fd3840bf6" containerName="revoke-keys"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316811    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bdb411e4-b75e-486b-a266-ee9fd3840bf6" containerName="revoke-keys"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316818    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8dce3c4c-8214-408d-9149-d096eff5ac7c" containerName="main"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316822    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="8dce3c4c-8214-408d-9149-d096eff5ac7c" containerName="main"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316828    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316833    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316891    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316898    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="070e50f4-8668-449d-af63-8dfe4436d432" containerName="main"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316904    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bbd6433c-735b-439f-b53b-8be16b6746c6" containerName="payload-pre-processing"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316908    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="8dce3c4c-8214-408d-9149-d096eff5ac7c" containerName="main"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316913    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="3be77961-a2fb-4439-b06e-d27ecf3ff8b0" containerName="payload-pre-processing"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316920    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy"
Sep 10 12:35:09.316938 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.316924    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bdb411e4-b75e-486b-a266-ee9fd3840bf6" containerName="revoke-keys"
Sep 10 12:35:09.321311 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.321293    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.324957 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.324936    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-b-a5be11-openshift-default-dockercfg-pf89r\""
Sep 10 12:35:09.342622 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.342590    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"]
Sep 10 12:35:09.429109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429078    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-envoy\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429114    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-token\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429137    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-24s9j\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-kube-api-access-24s9j\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429170    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-podinfo\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429187    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-data\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429220    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-socket\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429277    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-credential-socket\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429380    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-certs\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.429778 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.429494    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e99f80-e8da-4b4d-86b9-af22964a8828-istiod-ca-cert\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530435 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530406    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-socket\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530571 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530464    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-credential-socket\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530571 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530536    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-certs\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530668 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530585    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e99f80-e8da-4b4d-86b9-af22964a8828-istiod-ca-cert\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530668 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530624    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-envoy\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530668 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530652    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-token\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530677    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-24s9j\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-kube-api-access-24s9j\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530704    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-podinfo\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530728    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-data\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530960 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530816    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-socket\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530960 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530876    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-credential-socket\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.530960 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.530894    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-certs\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.531107 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.531039    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-data\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.531380 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.531359    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e99f80-e8da-4b4d-86b9-af22964a8828-istiod-ca-cert\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.533002 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.532974    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-envoy\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.533120 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.533101    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-podinfo\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.557092 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.557066    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-token\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.557314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.557298    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-24s9j\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-kube-api-access-24s9j\") pod \"e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") " pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.630677 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.630653    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:09.929709 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.929685    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"]
Sep 10 12:35:09.931180 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:35:09.931155    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd0e99f80_e8da_4b4d_86b9_af22964a8828.slice/crio-a188019e5d8023339add7332197777a70c17c239c6dbbf8fcab7875677cff336 WatchSource:0}: Error finding container a188019e5d8023339add7332197777a70c17c239c6dbbf8fcab7875677cff336: Status 404 returned error can't find the container with id a188019e5d8023339add7332197777a70c17c239c6dbbf8fcab7875677cff336
Sep 10 12:35:09.933273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.933243    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:35:09.933344 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.933308    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:35:09.933344 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:09.933338    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:35:10.506503 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:10.506405    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" event={"ID":"d0e99f80-e8da-4b4d-86b9-af22964a8828","Type":"ContainerStarted","Data":"806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67"}
Sep 10 12:35:10.506503 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:10.506485    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" event={"ID":"d0e99f80-e8da-4b4d-86b9-af22964a8828","Type":"ContainerStarted","Data":"a188019e5d8023339add7332197777a70c17c239c6dbbf8fcab7875677cff336"}
Sep 10 12:35:10.537285 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:10.537232    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" podStartSLOduration=1.5372154679999999 podStartE2EDuration="1.537215468s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:35:10.536922743 +0000 UTC m=+1521.733438442" watchObservedRunningTime="2026-09-10 12:35:10.537215468 +0000 UTC m=+1521.733731169"
Sep 10 12:35:10.631609 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:10.631579    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:10.636262 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:10.636240    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:11.511247 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:11.511207    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:11.512148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:11.512130    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:14.928037 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.928007    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"]
Sep 10 12:35:14.928614 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.928596    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="18e963c5-8aea-42d7-bcb5-71be9e9331b0" containerName="istio-proxy"
Sep 10 12:35:14.928712 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.928619    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="972fe1af-6eae-4789-8e38-6bc959916e86" containerName="istio-proxy"
Sep 10 12:35:14.931608 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.931589    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:14.937871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.937842    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-a5be11-dockercfg-w7tvc\""
Sep 10 12:35:14.937980 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.937888    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-a5be11\""
Sep 10 12:35:14.953900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.953878    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"]
Sep 10 12:35:14.974880 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.974859    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969fd257-5edf-4ef2-b83a-3d981dddcc19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:14.974978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:14.974904    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bv8pl\" (UniqueName: \"kubernetes.io/projected/969fd257-5edf-4ef2-b83a-3d981dddcc19-kube-api-access-bv8pl\") pod \"payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:15.038807 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.038774    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"]
Sep 10 12:35:15.042116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.042101    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.056717 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.056693    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"]
Sep 10 12:35:15.075951 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.075930    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eae2b2-17c1-4164-89ed-a7d1d7949505-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.076053 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.075967    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpn2n\" (UniqueName: \"kubernetes.io/projected/65eae2b2-17c1-4164-89ed-a7d1d7949505-kube-api-access-jpn2n\") pod \"payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.076053 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.076020    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969fd257-5edf-4ef2-b83a-3d981dddcc19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:15.076130 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.076083    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bv8pl\" (UniqueName: \"kubernetes.io/projected/969fd257-5edf-4ef2-b83a-3d981dddcc19-kube-api-access-bv8pl\") pod \"payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:15.076684 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.076668    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969fd257-5edf-4ef2-b83a-3d981dddcc19-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:15.089732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.089709    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bv8pl\" (UniqueName: \"kubernetes.io/projected/969fd257-5edf-4ef2-b83a-3d981dddcc19-kube-api-access-bv8pl\") pod \"payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:15.176525 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.176505    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eae2b2-17c1-4164-89ed-a7d1d7949505-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.176618 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.176538    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpn2n\" (UniqueName: \"kubernetes.io/projected/65eae2b2-17c1-4164-89ed-a7d1d7949505-kube-api-access-jpn2n\") pod \"payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.177104 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.177088    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eae2b2-17c1-4164-89ed-a7d1d7949505-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.186384 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.186340    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpn2n\" (UniqueName: \"kubernetes.io/projected/65eae2b2-17c1-4164-89ed-a7d1d7949505-kube-api-access-jpn2n\") pod \"payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.241156 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.241131    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:15.351526 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.351499    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:15.519990 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.519965    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"]
Sep 10 12:35:15.521684 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:35:15.521652    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod969fd257_5edf_4ef2_b83a_3d981dddcc19.slice/crio-f2a36e4304255994ccf1c3ecf5e9701831a1e23347c48d933bd58977cc8fb777 WatchSource:0}: Error finding container f2a36e4304255994ccf1c3ecf5e9701831a1e23347c48d933bd58977cc8fb777: Status 404 returned error can't find the container with id f2a36e4304255994ccf1c3ecf5e9701831a1e23347c48d933bd58977cc8fb777
Sep 10 12:35:15.532421 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.532397    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5" event={"ID":"969fd257-5edf-4ef2-b83a-3d981dddcc19","Type":"ContainerStarted","Data":"f2a36e4304255994ccf1c3ecf5e9701831a1e23347c48d933bd58977cc8fb777"}
Sep 10 12:35:15.630811 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:15.630784    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"]
Sep 10 12:35:15.631695 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:35:15.631672    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65eae2b2_17c1_4164_89ed_a7d1d7949505.slice/crio-c7b5ad7fe766655910ee99e972a165a4a2a6dff87fb370b1a874f9c889efc7c3 WatchSource:0}: Error finding container c7b5ad7fe766655910ee99e972a165a4a2a6dff87fb370b1a874f9c889efc7c3: Status 404 returned error can't find the container with id c7b5ad7fe766655910ee99e972a165a4a2a6dff87fb370b1a874f9c889efc7c3
Sep 10 12:35:16.539856 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:16.539820    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx" event={"ID":"65eae2b2-17c1-4164-89ed-a7d1d7949505","Type":"ContainerStarted","Data":"8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be"}
Sep 10 12:35:16.540307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:16.539920    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:16.540307 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:16.539948    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx" event={"ID":"65eae2b2-17c1-4164-89ed-a7d1d7949505","Type":"ContainerStarted","Data":"c7b5ad7fe766655910ee99e972a165a4a2a6dff87fb370b1a874f9c889efc7c3"}
Sep 10 12:35:16.541479 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:16.541454    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5" event={"ID":"969fd257-5edf-4ef2-b83a-3d981dddcc19","Type":"ContainerStarted","Data":"7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf"}
Sep 10 12:35:16.541584 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:16.541569    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:16.565736 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:16.565697    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx" podStartSLOduration=2.565682185 podStartE2EDuration="2.565682185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:35:16.565531858 +0000 UTC m=+1527.762047555" watchObservedRunningTime="2026-09-10 12:35:16.565682185 +0000 UTC m=+1527.762197889"
Sep 10 12:35:16.589244 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:16.589209    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5" podStartSLOduration=2.589197354 podStartE2EDuration="2.589197354s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:35:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:35:16.589021198 +0000 UTC m=+1527.785536896" watchObservedRunningTime="2026-09-10 12:35:16.589197354 +0000 UTC m=+1527.785713095"
Sep 10 12:35:20.051413 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:20.051376    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:35:27.549375 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:27.549343    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:35:27.549870 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:27.549473    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:35:29.605133 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:29.605059    2552 generic.go:356] "Generic (PLEG): container finished" podID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerID="806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67" exitCode=0
Sep 10 12:35:29.605530 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:29.605146    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" event={"ID":"d0e99f80-e8da-4b4d-86b9-af22964a8828","Type":"ContainerDied","Data":"806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67"}
Sep 10 12:35:29.605606 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:29.605547    2552 scope.go:117] "RemoveContainer" containerID="806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67"
Sep 10 12:35:29.606775 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:29.606739    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:35:29.606886 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:29.606825    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:35:29.606886 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:29.606870    2552 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892152Ki","pods":"250"}
Sep 10 12:35:30.614806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:30.614767    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" event={"ID":"d0e99f80-e8da-4b4d-86b9-af22964a8828","Type":"ContainerStarted","Data":"421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df"}
Sep 10 12:35:30.631474 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:30.631443    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:30.631644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:30.631609    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 12:35:30.631689 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:30.631655    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 12:35:31.631606 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:31.631573    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 12:35:31.631978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:31.631632    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 12:35:32.631673 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:32.631641    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 12:35:32.632078 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:32.631695    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 12:35:33.631622 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:33.631588    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 12:35:33.631800 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:33.631668    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 12:35:34.630997 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:34.630962    2552 patch_prober.go:28] interesting pod/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body=
Sep 10 12:35:34.631204 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:34.631030    2552 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused"
Sep 10 12:35:35.634533 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:35.634503    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:35.634896 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:35.634746    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:35.635284 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:35.635268    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:35:42.599041 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:42.599013    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx_65eae2b2-17c1-4164-89ed-a7d1d7949505/payload-processing/0.log"
Sep 10 12:35:49.529133 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.529103    2552 scope.go:117] "RemoveContainer" containerID="27863fa6940321223a0e22c9f0a2a0accf4837155459e57800e02d62a58b106b"
Sep 10 12:35:49.536881 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.536865    2552 scope.go:117] "RemoveContainer" containerID="ce8be83886395ac2d40ba27d7cce594dd235dbe9c559be46cba15470b9d2309e"
Sep 10 12:35:49.688826 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.688801    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"]
Sep 10 12:35:49.693556 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.693540    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.701371 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.701348    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3db7ea\"/\"ipp-route-3db7ea-kserve-self-signed-certs\""
Sep 10 12:35:49.701522 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.701394    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-3db7ea\"/\"default-dockercfg-rwrcc\""
Sep 10 12:35:49.701522 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.701396    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3db7ea\"/\"kube-root-ca.crt\""
Sep 10 12:35:49.701522 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.701396    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-3db7ea\"/\"openshift-service-ca.crt\""
Sep 10 12:35:49.709663 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.709643    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"]
Sep 10 12:35:49.766063 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.766039    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-home\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.766178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.766072    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-dshm\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.766178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.766094    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af580e9-3f13-4203-94fc-40385f2e6611-tls-certs\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.766178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.766149    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktkbn\" (UniqueName: \"kubernetes.io/projected/2af580e9-3f13-4203-94fc-40385f2e6611-kube-api-access-ktkbn\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.766314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.766197    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-tmp-dir\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.766314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.766220    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-model-cache\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.866723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.866692    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-dshm\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.866848 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.866730    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af580e9-3f13-4203-94fc-40385f2e6611-tls-certs\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.866848 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.866759    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktkbn\" (UniqueName: \"kubernetes.io/projected/2af580e9-3f13-4203-94fc-40385f2e6611-kube-api-access-ktkbn\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.866848 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.866796    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-tmp-dir\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.866848 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.866825    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-model-cache\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.867054 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.866915    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-home\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.867195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.867166    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-tmp-dir\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.867195 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.867192    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-home\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.867323 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.867245    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-model-cache\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.869149 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.869120    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-dshm\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.869301 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.869285    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af580e9-3f13-4203-94fc-40385f2e6611-tls-certs\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:49.876995 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:49.876973    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktkbn\" (UniqueName: \"kubernetes.io/projected/2af580e9-3f13-4203-94fc-40385f2e6611-kube-api-access-ktkbn\") pod \"ipp-route-3db7ea-kserve-77777d5f8c-zgzjl\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") " pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:50.004202 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:50.004179    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:35:50.276958 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:50.276890    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"]
Sep 10 12:35:50.279141 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:35:50.279113    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2af580e9_3f13_4203_94fc_40385f2e6611.slice/crio-b38dca00066ffe00ff35fa3bd43f8f69a03d99b55954e6f6f5311697721e5e79 WatchSource:0}: Error finding container b38dca00066ffe00ff35fa3bd43f8f69a03d99b55954e6f6f5311697721e5e79: Status 404 returned error can't find the container with id b38dca00066ffe00ff35fa3bd43f8f69a03d99b55954e6f6f5311697721e5e79
Sep 10 12:35:50.722057 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:50.722026    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl" event={"ID":"2af580e9-3f13-4203-94fc-40385f2e6611","Type":"ContainerStarted","Data":"5ea99ad7319d003a513e88c96fff161003ad0b7a45d33173a295475f5fea3bc9"}
Sep 10 12:35:50.722394 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:50.722069    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl" event={"ID":"2af580e9-3f13-4203-94fc-40385f2e6611","Type":"ContainerStarted","Data":"b38dca00066ffe00ff35fa3bd43f8f69a03d99b55954e6f6f5311697721e5e79"}
Sep 10 12:35:50.750580 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:50.750533    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl" podStartSLOduration=1.7505193430000001 podStartE2EDuration="1.750519343s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:35:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:35:50.749753611 +0000 UTC m=+1561.946269311" watchObservedRunningTime="2026-09-10 12:35:50.750519343 +0000 UTC m=+1561.947035081"
Sep 10 12:35:53.353206 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:35:53.353168    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:36:00.005012 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:00.004981    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:36:00.005518 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:00.005025    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:36:00.016676 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:00.016656    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:36:00.776804 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:00.776780    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:36:12.155798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:12.155767    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:36:18.483514 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:18.483482    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx_65eae2b2-17c1-4164-89ed-a7d1d7949505/payload-processing/0.log"
Sep 10 12:36:25.078697 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.078662    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"]
Sep 10 12:36:25.082212 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.082193    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.086706 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.086685    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-47deee-dockercfg-gccc7\""
Sep 10 12:36:25.092578 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.092557    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-47deee\""
Sep 10 12:36:25.133537 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.133515    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58vvw\" (UniqueName: \"kubernetes.io/projected/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-kube-api-access-58vvw\") pod \"payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.133653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.133543    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.152877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.152850    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"]
Sep 10 12:36:25.234989 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.234959    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58vvw\" (UniqueName: \"kubernetes.io/projected/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-kube-api-access-58vvw\") pod \"payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.234989 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.234991    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.235676 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.235658    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.252219 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.252198    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-58vvw\" (UniqueName: \"kubernetes.io/projected/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-kube-api-access-58vvw\") pod \"payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.392779 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.392758    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.677908 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.677837    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"]
Sep 10 12:36:25.681106 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:36:25.681083    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc1b95b2_dc1c_43ff_b49b_1e8b7097f2b9.slice/crio-4159f3b6ec0980bccf9571305e6604c087e909df336f4919afb354581394200f WatchSource:0}: Error finding container 4159f3b6ec0980bccf9571305e6604c087e909df336f4919afb354581394200f: Status 404 returned error can't find the container with id 4159f3b6ec0980bccf9571305e6604c087e909df336f4919afb354581394200f
Sep 10 12:36:25.880779 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.880747    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2" event={"ID":"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9","Type":"ContainerStarted","Data":"ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e"}
Sep 10 12:36:25.880937 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.880804    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2" event={"ID":"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9","Type":"ContainerStarted","Data":"4159f3b6ec0980bccf9571305e6604c087e909df336f4919afb354581394200f"}
Sep 10 12:36:25.880937 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.880848    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:25.922671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:25.922627    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2" podStartSLOduration=0.922613445 podStartE2EDuration="922.613445ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:36:25.920383974 +0000 UTC m=+1597.116899683" watchObservedRunningTime="2026-09-10 12:36:25.922613445 +0000 UTC m=+1597.119129142"
Sep 10 12:36:31.665708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:31.665676    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:36:36.887462 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:36.887411    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:49.570512 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:49.570452    2552 scope.go:117] "RemoveContainer" containerID="a9be319146f76cdb727ccc6cf9a3e598d14ce579419fb4c31253d879f4ec49e5"
Sep 10 12:36:49.578251 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:49.578235    2552 scope.go:117] "RemoveContainer" containerID="0bdba2d15c547b8e547748e764edbf74669a705879a9a09d0b4dd064e7fa52ac"
Sep 10 12:36:54.176365 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.176333    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"]
Sep 10 12:36:54.176850 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.176566    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2" podUID="bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" containerName="payload-processing" containerID="cri-o://ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e" gracePeriod=30
Sep 10 12:36:54.417576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.417550    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:54.572269 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.572193    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-58vvw\" (UniqueName: \"kubernetes.io/projected/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-kube-api-access-58vvw\") pod \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") "
Sep 10 12:36:54.572269 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.572264    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-plugins-config-volume\") pod \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\" (UID: \"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9\") "
Sep 10 12:36:54.572645 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.572613    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" (UID: "bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:36:54.574308 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.574286    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-kube-api-access-58vvw" (OuterVolumeSpecName: "kube-api-access-58vvw") pod "bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" (UID: "bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9"). InnerVolumeSpecName "kube-api-access-58vvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:36:54.673632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.673601    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58vvw\" (UniqueName: \"kubernetes.io/projected/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-kube-api-access-58vvw\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:36:54.673632 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:54.673630    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:36:55.015843 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.015816    2552 generic.go:356] "Generic (PLEG): container finished" podID="bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" containerID="ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e" exitCode=0
Sep 10 12:36:55.015965 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.015855    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"
Sep 10 12:36:55.015965 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.015899    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2" event={"ID":"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9","Type":"ContainerDied","Data":"ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e"}
Sep 10 12:36:55.015965 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.015948    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2" event={"ID":"bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9","Type":"ContainerDied","Data":"4159f3b6ec0980bccf9571305e6604c087e909df336f4919afb354581394200f"}
Sep 10 12:36:55.016087 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.015968    2552 scope.go:117] "RemoveContainer" containerID="ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e"
Sep 10 12:36:55.024373 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.024358    2552 scope.go:117] "RemoveContainer" containerID="ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e"
Sep 10 12:36:55.024682 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:36:55.024668    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e\": container with ID starting with ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e not found: ID does not exist" containerID="ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e"
Sep 10 12:36:55.024750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.024689    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e"} err="failed to get container status \"ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e\": rpc error: code = NotFound desc = could not find container \"ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e\": container with ID starting with ccc3ed3ceaa8649ed8700b3227d8fc6245b62239d05136e52f12c43e4f70354e not found: ID does not exist"
Sep 10 12:36:55.043807 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.043785    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"]
Sep 10 12:36:55.046233 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.046213    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-47deee-577b6bd8f5-qnxg2"]
Sep 10 12:36:55.242269 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:55.242242    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" path="/var/lib/kubelet/pods/bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9/volumes"
Sep 10 12:36:58.366362 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:36:58.366334    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:37:18.875348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:18.875316    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:37:26.969001 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:26.968973    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"]
Sep 10 12:37:32.146527 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:32.146491    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:37:41.988155 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:41.988092    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl" podUID="2af580e9-3f13-4203-94fc-40385f2e6611" containerName="main" containerID="cri-o://5ea99ad7319d003a513e88c96fff161003ad0b7a45d33173a295475f5fea3bc9" gracePeriod=45
Sep 10 12:37:42.230613 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.230578    2552 generic.go:356] "Generic (PLEG): container finished" podID="2af580e9-3f13-4203-94fc-40385f2e6611" containerID="5ea99ad7319d003a513e88c96fff161003ad0b7a45d33173a295475f5fea3bc9" exitCode=0
Sep 10 12:37:42.230724 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.230617    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl" event={"ID":"2af580e9-3f13-4203-94fc-40385f2e6611","Type":"ContainerDied","Data":"5ea99ad7319d003a513e88c96fff161003ad0b7a45d33173a295475f5fea3bc9"}
Sep 10 12:37:42.230724 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.230664    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl" event={"ID":"2af580e9-3f13-4203-94fc-40385f2e6611","Type":"ContainerDied","Data":"b38dca00066ffe00ff35fa3bd43f8f69a03d99b55954e6f6f5311697721e5e79"}
Sep 10 12:37:42.230724 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.230675    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b38dca00066ffe00ff35fa3bd43f8f69a03d99b55954e6f6f5311697721e5e79"
Sep 10 12:37:42.236170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.236153    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:37:42.358860 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.358794    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-model-cache\") pod \"2af580e9-3f13-4203-94fc-40385f2e6611\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") "
Sep 10 12:37:42.358860 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.358827    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af580e9-3f13-4203-94fc-40385f2e6611-tls-certs\") pod \"2af580e9-3f13-4203-94fc-40385f2e6611\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") "
Sep 10 12:37:42.359046 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.358864    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-tmp-dir\") pod \"2af580e9-3f13-4203-94fc-40385f2e6611\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") "
Sep 10 12:37:42.359046 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.358910    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-home\") pod \"2af580e9-3f13-4203-94fc-40385f2e6611\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") "
Sep 10 12:37:42.359046 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.358984    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ktkbn\" (UniqueName: \"kubernetes.io/projected/2af580e9-3f13-4203-94fc-40385f2e6611-kube-api-access-ktkbn\") pod \"2af580e9-3f13-4203-94fc-40385f2e6611\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") "
Sep 10 12:37:42.359046 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.359011    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-dshm\") pod \"2af580e9-3f13-4203-94fc-40385f2e6611\" (UID: \"2af580e9-3f13-4203-94fc-40385f2e6611\") "
Sep 10 12:37:42.359255 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.359039    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-model-cache" (OuterVolumeSpecName: "model-cache") pod "2af580e9-3f13-4203-94fc-40385f2e6611" (UID: "2af580e9-3f13-4203-94fc-40385f2e6611"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:37:42.359255 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.359139    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "2af580e9-3f13-4203-94fc-40385f2e6611" (UID: "2af580e9-3f13-4203-94fc-40385f2e6611"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:37:42.359348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.359254    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-home" (OuterVolumeSpecName: "home") pod "2af580e9-3f13-4203-94fc-40385f2e6611" (UID: "2af580e9-3f13-4203-94fc-40385f2e6611"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:37:42.359348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.359266    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:37:42.359348 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.359309    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:37:42.361075 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.361049    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-dshm" (OuterVolumeSpecName: "dshm") pod "2af580e9-3f13-4203-94fc-40385f2e6611" (UID: "2af580e9-3f13-4203-94fc-40385f2e6611"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:37:42.361166 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.361075    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2af580e9-3f13-4203-94fc-40385f2e6611-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "2af580e9-3f13-4203-94fc-40385f2e6611" (UID: "2af580e9-3f13-4203-94fc-40385f2e6611"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:37:42.361166 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.361138    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2af580e9-3f13-4203-94fc-40385f2e6611-kube-api-access-ktkbn" (OuterVolumeSpecName: "kube-api-access-ktkbn") pod "2af580e9-3f13-4203-94fc-40385f2e6611" (UID: "2af580e9-3f13-4203-94fc-40385f2e6611"). InnerVolumeSpecName "kube-api-access-ktkbn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:37:42.460686 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.460661    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:37:42.460686 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.460684    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ktkbn\" (UniqueName: \"kubernetes.io/projected/2af580e9-3f13-4203-94fc-40385f2e6611-kube-api-access-ktkbn\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:37:42.460816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.460694    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/2af580e9-3f13-4203-94fc-40385f2e6611-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:37:42.460816 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:42.460702    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/2af580e9-3f13-4203-94fc-40385f2e6611-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:37:43.236363 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:43.236333    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"
Sep 10 12:37:43.258858 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:43.258832    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"]
Sep 10 12:37:43.262653 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:43.262632    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-3db7ea/ipp-route-3db7ea-kserve-77777d5f8c-zgzjl"]
Sep 10 12:37:45.241873 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:37:45.241840    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2af580e9-3f13-4203-94fc-40385f2e6611" path="/var/lib/kubelet/pods/2af580e9-3f13-4203-94fc-40385f2e6611/volumes"
Sep 10 12:38:21.883017 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:21.882935    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"]
Sep 10 12:38:21.883613 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:21.883134    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx" podUID="65eae2b2-17c1-4164-89ed-a7d1d7949505" containerName="payload-processing" containerID="cri-o://8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be" gracePeriod=30
Sep 10 12:38:21.892670 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:21.892647    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"]
Sep 10 12:38:21.892837 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:21.892813    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5" podUID="969fd257-5edf-4ef2-b83a-3d981dddcc19" containerName="payload-pre-processing" containerID="cri-o://7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf" gracePeriod=30
Sep 10 12:38:22.129661 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.129641    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:38:22.153078 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.153008    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:38:22.180579 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.180551    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bv8pl\" (UniqueName: \"kubernetes.io/projected/969fd257-5edf-4ef2-b83a-3d981dddcc19-kube-api-access-bv8pl\") pod \"969fd257-5edf-4ef2-b83a-3d981dddcc19\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") "
Sep 10 12:38:22.180690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.180599    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eae2b2-17c1-4164-89ed-a7d1d7949505-plugins-config-volume\") pod \"65eae2b2-17c1-4164-89ed-a7d1d7949505\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") "
Sep 10 12:38:22.180731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.180689    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jpn2n\" (UniqueName: \"kubernetes.io/projected/65eae2b2-17c1-4164-89ed-a7d1d7949505-kube-api-access-jpn2n\") pod \"65eae2b2-17c1-4164-89ed-a7d1d7949505\" (UID: \"65eae2b2-17c1-4164-89ed-a7d1d7949505\") "
Sep 10 12:38:22.180731 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.180724    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969fd257-5edf-4ef2-b83a-3d981dddcc19-plugins-config-volume\") pod \"969fd257-5edf-4ef2-b83a-3d981dddcc19\" (UID: \"969fd257-5edf-4ef2-b83a-3d981dddcc19\") "
Sep 10 12:38:22.181047 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.181018    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65eae2b2-17c1-4164-89ed-a7d1d7949505-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "65eae2b2-17c1-4164-89ed-a7d1d7949505" (UID: "65eae2b2-17c1-4164-89ed-a7d1d7949505"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:38:22.181331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.181155    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/969fd257-5edf-4ef2-b83a-3d981dddcc19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "969fd257-5edf-4ef2-b83a-3d981dddcc19" (UID: "969fd257-5edf-4ef2-b83a-3d981dddcc19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:38:22.182625 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.182588    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/969fd257-5edf-4ef2-b83a-3d981dddcc19-kube-api-access-bv8pl" (OuterVolumeSpecName: "kube-api-access-bv8pl") pod "969fd257-5edf-4ef2-b83a-3d981dddcc19" (UID: "969fd257-5edf-4ef2-b83a-3d981dddcc19"). InnerVolumeSpecName "kube-api-access-bv8pl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:22.182768 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.182753    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65eae2b2-17c1-4164-89ed-a7d1d7949505-kube-api-access-jpn2n" (OuterVolumeSpecName: "kube-api-access-jpn2n") pod "65eae2b2-17c1-4164-89ed-a7d1d7949505" (UID: "65eae2b2-17c1-4164-89ed-a7d1d7949505"). InnerVolumeSpecName "kube-api-access-jpn2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:22.281491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.281456    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpn2n\" (UniqueName: \"kubernetes.io/projected/65eae2b2-17c1-4164-89ed-a7d1d7949505-kube-api-access-jpn2n\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:22.281491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.281486    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/969fd257-5edf-4ef2-b83a-3d981dddcc19-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:22.281736 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.281501    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bv8pl\" (UniqueName: \"kubernetes.io/projected/969fd257-5edf-4ef2-b83a-3d981dddcc19-kube-api-access-bv8pl\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:22.281736 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.281513    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/65eae2b2-17c1-4164-89ed-a7d1d7949505-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:22.416246 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.416170    2552 generic.go:356] "Generic (PLEG): container finished" podID="65eae2b2-17c1-4164-89ed-a7d1d7949505" containerID="8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be" exitCode=0
Sep 10 12:38:22.416246 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.416223    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"
Sep 10 12:38:22.416498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.416254    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx" event={"ID":"65eae2b2-17c1-4164-89ed-a7d1d7949505","Type":"ContainerDied","Data":"8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be"}
Sep 10 12:38:22.416498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.416312    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx" event={"ID":"65eae2b2-17c1-4164-89ed-a7d1d7949505","Type":"ContainerDied","Data":"c7b5ad7fe766655910ee99e972a165a4a2a6dff87fb370b1a874f9c889efc7c3"}
Sep 10 12:38:22.416498 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.416345    2552 scope.go:117] "RemoveContainer" containerID="8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be"
Sep 10 12:38:22.417612 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.417589    2552 generic.go:356] "Generic (PLEG): container finished" podID="969fd257-5edf-4ef2-b83a-3d981dddcc19" containerID="7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf" exitCode=0
Sep 10 12:38:22.417710 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.417635    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"
Sep 10 12:38:22.417710 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.417676    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5" event={"ID":"969fd257-5edf-4ef2-b83a-3d981dddcc19","Type":"ContainerDied","Data":"7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf"}
Sep 10 12:38:22.417900 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.417824    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5" event={"ID":"969fd257-5edf-4ef2-b83a-3d981dddcc19","Type":"ContainerDied","Data":"f2a36e4304255994ccf1c3ecf5e9701831a1e23347c48d933bd58977cc8fb777"}
Sep 10 12:38:22.425827 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.425805    2552 scope.go:117] "RemoveContainer" containerID="8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be"
Sep 10 12:38:22.426147 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:38:22.426124    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be\": container with ID starting with 8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be not found: ID does not exist" containerID="8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be"
Sep 10 12:38:22.426245 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.426152    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be"} err="failed to get container status \"8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be\": rpc error: code = NotFound desc = could not find container \"8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be\": container with ID starting with 8b10af5be962359fcf7e67c0300aee3a06d616d522959d20ba4538e69128e3be not found: ID does not exist"
Sep 10 12:38:22.426245 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.426167    2552 scope.go:117] "RemoveContainer" containerID="7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf"
Sep 10 12:38:22.433862 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.433848    2552 scope.go:117] "RemoveContainer" containerID="7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf"
Sep 10 12:38:22.434110 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:38:22.434090    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf\": container with ID starting with 7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf not found: ID does not exist" containerID="7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf"
Sep 10 12:38:22.434170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.434113    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf"} err="failed to get container status \"7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf\": rpc error: code = NotFound desc = could not find container \"7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf\": container with ID starting with 7685c7ee8f5bf82dfe3fa6df0e4a88ff91e0eb6d0b87aa59946387dc81a04edf not found: ID does not exist"
Sep 10 12:38:22.441036 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.441007    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"]
Sep 10 12:38:22.444360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.444341    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-a5be11-5fd949fb86-6kjb5"]
Sep 10 12:38:22.456164 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.456145    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"]
Sep 10 12:38:22.457583 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:22.457565    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-a5be11-686bfb7d89-482zx"]
Sep 10 12:38:23.241772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:23.241701    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="65eae2b2-17c1-4164-89ed-a7d1d7949505" path="/var/lib/kubelet/pods/65eae2b2-17c1-4164-89ed-a7d1d7949505/volumes"
Sep 10 12:38:23.242114 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:23.242025    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="969fd257-5edf-4ef2-b83a-3d981dddcc19" path="/var/lib/kubelet/pods/969fd257-5edf-4ef2-b83a-3d981dddcc19/volumes"
Sep 10 12:38:24.843109 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:24.843071    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:38:28.898250 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:28.898219    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"]
Sep 10 12:38:28.898732 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:28.898501    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy" containerID="cri-o://421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df" gracePeriod=30
Sep 10 12:38:34.136303 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.136275    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:38:34.179283 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179224    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-envoy\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179283 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179272    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-credential-socket\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179295    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-data\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179346    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-24s9j\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-kube-api-access-24s9j\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179405    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-socket\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179467 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179461    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e99f80-e8da-4b4d-86b9-af22964a8828-istiod-ca-cert\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179495    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-certs\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179541    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-podinfo\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179565    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-token\") pod \"d0e99f80-e8da-4b4d-86b9-af22964a8828\" (UID: \"d0e99f80-e8da-4b4d-86b9-af22964a8828\") "
Sep 10 12:38:34.179690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179594    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:38:34.179690 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179625    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-data" (OuterVolumeSpecName: "istio-data") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:38:34.179949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179786    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:38:34.179949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179873    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:38:34.179949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179896    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.179949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179915    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.179949 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.179929    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.180136 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.180009    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d0e99f80-e8da-4b4d-86b9-af22964a8828-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:38:34.181521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.181499    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:38:34.181718 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.181700    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-kube-api-access-24s9j" (OuterVolumeSpecName: "kube-api-access-24s9j") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "kube-api-access-24s9j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:34.181999 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.181977    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-token" (OuterVolumeSpecName: "istio-token") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:34.182184 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.182165    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "d0e99f80-e8da-4b4d-86b9-af22964a8828" (UID: "d0e99f80-e8da-4b4d-86b9-af22964a8828"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:38:34.280891 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.280867    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.280891 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.280888    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d0e99f80-e8da-4b4d-86b9-af22964a8828-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.281019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.280898    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.281019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.280905    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.281019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.280911    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d0e99f80-e8da-4b4d-86b9-af22964a8828-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.281019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.280920    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-24s9j\" (UniqueName: \"kubernetes.io/projected/d0e99f80-e8da-4b4d-86b9-af22964a8828-kube-api-access-24s9j\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:34.474024 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.473968    2552 generic.go:356] "Generic (PLEG): container finished" podID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerID="421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df" exitCode=0
Sep 10 12:38:34.474024 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.474016    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" event={"ID":"d0e99f80-e8da-4b4d-86b9-af22964a8828","Type":"ContainerDied","Data":"421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df"}
Sep 10 12:38:34.474143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.474054    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7" event={"ID":"d0e99f80-e8da-4b4d-86b9-af22964a8828","Type":"ContainerDied","Data":"a188019e5d8023339add7332197777a70c17c239c6dbbf8fcab7875677cff336"}
Sep 10 12:38:34.474143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.474074    2552 scope.go:117] "RemoveContainer" containerID="421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df"
Sep 10 12:38:34.474143 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.474022    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"
Sep 10 12:38:34.482570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.482556    2552 scope.go:117] "RemoveContainer" containerID="806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67"
Sep 10 12:38:34.490035 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.490021    2552 scope.go:117] "RemoveContainer" containerID="421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df"
Sep 10 12:38:34.490285 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:38:34.490266    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df\": container with ID starting with 421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df not found: ID does not exist" containerID="421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df"
Sep 10 12:38:34.490342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.490291    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df"} err="failed to get container status \"421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df\": rpc error: code = NotFound desc = could not find container \"421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df\": container with ID starting with 421f4cbba00d4f285d7d19d036b54990aa602a0098aded6bd8f9f1a8c55f44df not found: ID does not exist"
Sep 10 12:38:34.490342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.490307    2552 scope.go:117] "RemoveContainer" containerID="806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67"
Sep 10 12:38:34.490557 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:38:34.490543    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67\": container with ID starting with 806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67 not found: ID does not exist" containerID="806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67"
Sep 10 12:38:34.490608 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.490561    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67"} err="failed to get container status \"806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67\": rpc error: code = NotFound desc = could not find container \"806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67\": container with ID starting with 806e0173bad1ce440f961cf419c7ce87d2f3b141f3982461edcbf5c6ee6fcc67 not found: ID does not exist"
Sep 10 12:38:34.498176 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.498155    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"]
Sep 10 12:38:34.502043 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:34.502027    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-b-a5be11-openshift-default-697f77d946-bpwv7"]
Sep 10 12:38:35.242070 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:35.242044    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" path="/var/lib/kubelet/pods/d0e99f80-e8da-4b4d-86b9-af22964a8828/volumes"
Sep 10 12:38:42.642486 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:42.642454    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:38:49.612695 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:49.612668    2552 scope.go:117] "RemoveContainer" containerID="5c1da28b8df3ea2490d6d5012c547594ac59043f01c4f2e7213148a6cf58e18c"
Sep 10 12:38:57.854798 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:57.854714    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"]
Sep 10 12:38:57.856513 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:57.856491    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-f89a79-29817390-k7gwk"]
Sep 10 12:38:57.879089 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:57.879059    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"]
Sep 10 12:38:57.879359 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:57.879334    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh" podUID="a510b0fa-6547-41b0-9cfd-27696f31d1d1" containerName="payload-processing" containerID="cri-o://7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46" gracePeriod=30
Sep 10 12:38:57.893708 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:57.893682    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"]
Sep 10 12:38:57.893866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:57.893846    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq" podUID="7f1626bd-1dff-40a7-addd-d6d72ceb800c" containerName="payload-pre-processing" containerID="cri-o://bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991" gracePeriod=30
Sep 10 12:38:58.157757 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.157736    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:38:58.160516 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.160497    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:38:58.272095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.272066    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pkfl4\" (UniqueName: \"kubernetes.io/projected/7f1626bd-1dff-40a7-addd-d6d72ceb800c-kube-api-access-pkfl4\") pod \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") "
Sep 10 12:38:58.272095 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.272100    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a510b0fa-6547-41b0-9cfd-27696f31d1d1-plugins-config-volume\") pod \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") "
Sep 10 12:38:58.272316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.272151    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1626bd-1dff-40a7-addd-d6d72ceb800c-plugins-config-volume\") pod \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\" (UID: \"7f1626bd-1dff-40a7-addd-d6d72ceb800c\") "
Sep 10 12:38:58.272316 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.272224    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hsmdm\" (UniqueName: \"kubernetes.io/projected/a510b0fa-6547-41b0-9cfd-27696f31d1d1-kube-api-access-hsmdm\") pod \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\" (UID: \"a510b0fa-6547-41b0-9cfd-27696f31d1d1\") "
Sep 10 12:38:58.272574 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.272550    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f1626bd-1dff-40a7-addd-d6d72ceb800c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "7f1626bd-1dff-40a7-addd-d6d72ceb800c" (UID: "7f1626bd-1dff-40a7-addd-d6d72ceb800c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:38:58.272643 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.272556    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a510b0fa-6547-41b0-9cfd-27696f31d1d1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a510b0fa-6547-41b0-9cfd-27696f31d1d1" (UID: "a510b0fa-6547-41b0-9cfd-27696f31d1d1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:38:58.274282 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.274257    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f1626bd-1dff-40a7-addd-d6d72ceb800c-kube-api-access-pkfl4" (OuterVolumeSpecName: "kube-api-access-pkfl4") pod "7f1626bd-1dff-40a7-addd-d6d72ceb800c" (UID: "7f1626bd-1dff-40a7-addd-d6d72ceb800c"). InnerVolumeSpecName "kube-api-access-pkfl4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:58.274377 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.274313    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a510b0fa-6547-41b0-9cfd-27696f31d1d1-kube-api-access-hsmdm" (OuterVolumeSpecName: "kube-api-access-hsmdm") pod "a510b0fa-6547-41b0-9cfd-27696f31d1d1" (UID: "a510b0fa-6547-41b0-9cfd-27696f31d1d1"). InnerVolumeSpecName "kube-api-access-hsmdm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:38:58.373828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.373799    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hsmdm\" (UniqueName: \"kubernetes.io/projected/a510b0fa-6547-41b0-9cfd-27696f31d1d1-kube-api-access-hsmdm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:58.373828 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.373824    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pkfl4\" (UniqueName: \"kubernetes.io/projected/7f1626bd-1dff-40a7-addd-d6d72ceb800c-kube-api-access-pkfl4\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:58.373954 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.373840    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a510b0fa-6547-41b0-9cfd-27696f31d1d1-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:58.373954 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.373851    2552 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7f1626bd-1dff-40a7-addd-d6d72ceb800c-plugins-config-volume\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:38:58.589458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.589375    2552 generic.go:356] "Generic (PLEG): container finished" podID="7f1626bd-1dff-40a7-addd-d6d72ceb800c" containerID="bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991" exitCode=0
Sep 10 12:38:58.589458 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.589452    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"
Sep 10 12:38:58.589618 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.589468    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq" event={"ID":"7f1626bd-1dff-40a7-addd-d6d72ceb800c","Type":"ContainerDied","Data":"bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991"}
Sep 10 12:38:58.589618 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.589515    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq" event={"ID":"7f1626bd-1dff-40a7-addd-d6d72ceb800c","Type":"ContainerDied","Data":"39e0203a2b6fa69285bff264a384022b2c543a6c2097f795b1e2604cddf8dc53"}
Sep 10 12:38:58.589618 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.589536    2552 scope.go:117] "RemoveContainer" containerID="bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991"
Sep 10 12:38:58.590923 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.590906    2552 generic.go:356] "Generic (PLEG): container finished" podID="a510b0fa-6547-41b0-9cfd-27696f31d1d1" containerID="7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46" exitCode=0
Sep 10 12:38:58.591018 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.590957    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"
Sep 10 12:38:58.591018 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.590959    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh" event={"ID":"a510b0fa-6547-41b0-9cfd-27696f31d1d1","Type":"ContainerDied","Data":"7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46"}
Sep 10 12:38:58.591018 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.591003    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh" event={"ID":"a510b0fa-6547-41b0-9cfd-27696f31d1d1","Type":"ContainerDied","Data":"0633e7279d373cb1220879f91df71d308e2ca16002ff698058d1b9ebed838e22"}
Sep 10 12:38:58.598977 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.598962    2552 scope.go:117] "RemoveContainer" containerID="bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991"
Sep 10 12:38:58.599306 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:38:58.599276    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991\": container with ID starting with bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991 not found: ID does not exist" containerID="bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991"
Sep 10 12:38:58.599561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.599315    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991"} err="failed to get container status \"bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991\": rpc error: code = NotFound desc = could not find container \"bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991\": container with ID starting with bf4e71643e250e606a1b5042c8ac73e5eb97cbb5d9d26f7d21419bd56f202991 not found: ID does not exist"
Sep 10 12:38:58.599561 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.599338    2552 scope.go:117] "RemoveContainer" containerID="7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46"
Sep 10 12:38:58.607124 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.607108    2552 scope.go:117] "RemoveContainer" containerID="7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46"
Sep 10 12:38:58.607390 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:38:58.607373    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46\": container with ID starting with 7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46 not found: ID does not exist" containerID="7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46"
Sep 10 12:38:58.607464 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.607397    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46"} err="failed to get container status \"7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46\": rpc error: code = NotFound desc = could not find container \"7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46\": container with ID starting with 7d5622393c507be65940b435a4a20e219d80f48a5d7b48dcec38938cfbf40b46 not found: ID does not exist"
Sep 10 12:38:58.614034 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.614015    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"]
Sep 10 12:38:58.617909 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.617891    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-f89a79-986569f89-hmzhh"]
Sep 10 12:38:58.627604 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.627585    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"]
Sep 10 12:38:58.633360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:58.633343    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-f89a79-59f7d86f74-8mbnq"]
Sep 10 12:38:59.242079 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:59.242053    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7f1626bd-1dff-40a7-addd-d6d72ceb800c" path="/var/lib/kubelet/pods/7f1626bd-1dff-40a7-addd-d6d72ceb800c/volumes"
Sep 10 12:38:59.242446 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:59.242292    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8507d204-52b0-487b-9d06-2af3dc3aeaaa" path="/var/lib/kubelet/pods/8507d204-52b0-487b-9d06-2af3dc3aeaaa/volumes"
Sep 10 12:38:59.242569 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:38:59.242557    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a510b0fa-6547-41b0-9cfd-27696f31d1d1" path="/var/lib/kubelet/pods/a510b0fa-6547-41b0-9cfd-27696f31d1d1/volumes"
Sep 10 12:39:00.349488 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:00.347491    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:39:04.759391 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:04.759361    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"]
Sep 10 12:39:04.759788 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:04.759575    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy" containerID="cri-o://4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874" gracePeriod=30
Sep 10 12:39:08.978360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:08.978330    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:39:10.002797 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.002773    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:39:10.157160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157140    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-podinfo\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157284 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157172    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-data\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157284 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157214    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7jfjd\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-kube-api-access-7jfjd\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157284 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157244    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-credential-socket\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157392 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157292    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-token\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157392 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157318    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3e800d-e06e-4251-8dc7-77573950ade0-istiod-ca-cert\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157392 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157364    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-envoy\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157397    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-socket\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157464    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-certs\") pod \"0b3e800d-e06e-4251-8dc7-77573950ade0\" (UID: \"0b3e800d-e06e-4251-8dc7-77573950ade0\") "
Sep 10 12:39:10.157570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157512    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-data" (OuterVolumeSpecName: "istio-data") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:39:10.157570 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157541    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:39:10.157776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157691    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:39:10.157776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157729    2552 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-data\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.157776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157737    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b3e800d-e06e-4251-8dc7-77573950ade0-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:39:10.157776 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157748    2552 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-credential-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.157947 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.157824    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:39:10.159399 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.159376    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 10 12:39:10.159532 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.159487    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:39:10.159777 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.159759    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-kube-api-access-7jfjd" (OuterVolumeSpecName: "kube-api-access-7jfjd") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "kube-api-access-7jfjd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:39:10.159838 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.159806    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-token" (OuterVolumeSpecName: "istio-token") pod "0b3e800d-e06e-4251-8dc7-77573950ade0" (UID: "0b3e800d-e06e-4251-8dc7-77573950ade0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:39:10.259039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.259014    2552 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-socket\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.259039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.259036    2552 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-workload-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.259160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.259045    2552 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-podinfo\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.259160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.259053    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jfjd\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-kube-api-access-7jfjd\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.259160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.259063    2552 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-token\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.259160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.259071    2552 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0b3e800d-e06e-4251-8dc7-77573950ade0-istiod-ca-cert\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.259160 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.259078    2552 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0b3e800d-e06e-4251-8dc7-77573950ade0-istio-envoy\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:39:10.647049 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.647027    2552 generic.go:356] "Generic (PLEG): container finished" podID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerID="4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874" exitCode=0
Sep 10 12:39:10.647179 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.647082    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"
Sep 10 12:39:10.647179 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.647118    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" event={"ID":"0b3e800d-e06e-4251-8dc7-77573950ade0","Type":"ContainerDied","Data":"4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874"}
Sep 10 12:39:10.647179 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.647174    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf" event={"ID":"0b3e800d-e06e-4251-8dc7-77573950ade0","Type":"ContainerDied","Data":"69b08a2242e31d2a28a2888e5bcb410563bcfd4bc3ebe88634dbeaf19c68f30e"}
Sep 10 12:39:10.647345 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.647199    2552 scope.go:117] "RemoveContainer" containerID="4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874"
Sep 10 12:39:10.655927 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.655912    2552 scope.go:117] "RemoveContainer" containerID="62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644"
Sep 10 12:39:10.663234 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.663219    2552 scope.go:117] "RemoveContainer" containerID="4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874"
Sep 10 12:39:10.663502 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:39:10.663486    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874\": container with ID starting with 4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874 not found: ID does not exist" containerID="4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874"
Sep 10 12:39:10.663585 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.663510    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874"} err="failed to get container status \"4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874\": rpc error: code = NotFound desc = could not find container \"4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874\": container with ID starting with 4fd1cba0c1c5a122b8b162c4401eacee6508e650042a2d61d9e1649fbb970874 not found: ID does not exist"
Sep 10 12:39:10.663585 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.663525    2552 scope.go:117] "RemoveContainer" containerID="62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644"
Sep 10 12:39:10.663753 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:39:10.663739    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644\": container with ID starting with 62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644 not found: ID does not exist" containerID="62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644"
Sep 10 12:39:10.663790 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.663757    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644"} err="failed to get container status \"62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644\": rpc error: code = NotFound desc = could not find container \"62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644\": container with ID starting with 62365f4da00c5c507f39a44752575b67cc94daa4aeef8eb6c129d2e993f4b644 not found: ID does not exist"
Sep 10 12:39:10.670750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.670730    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"]
Sep 10 12:39:10.676025 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:10.676008    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-f89a79-openshift-default-74466b87cd-g5ncf"]
Sep 10 12:39:11.241935 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:11.241907    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" path="/var/lib/kubelet/pods/0b3e800d-e06e-4251-8dc7-77573950ade0/volumes"
Sep 10 12:39:49.640399 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:49.639656    2552 scope.go:117] "RemoveContainer" containerID="b571b34ee7d0e623114c64fffd1c6c1039625c43e4bfe0f68bbd25b2ff2b3eb2"
Sep 10 12:39:49.651738 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:39:49.651712    2552 scope.go:117] "RemoveContainer" containerID="64ce06c54b8c6103aa3e3e6635717fac1210bff4294293da7c136e07593a1bbb"
Sep 10 12:40:30.094048 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:40:30.094013    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:40:49.682254 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:40:49.682230    2552 scope.go:117] "RemoveContainer" containerID="79775e4bb390334bb282f90f3a8d8cefc140fe91519b55e84ca1301104b1bcf9"
Sep 10 12:40:57.153800 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:40:57.153764    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:41:04.770762 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:41:04.770721    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:41:11.452980 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:41:11.452907    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:41:28.554383 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:41:28.554346    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:41:34.847871 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:41:34.847836    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:42:49.704033 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:42:49.703949    2552 scope.go:117] "RemoveContainer" containerID="5ea99ad7319d003a513e88c96fff161003ad0b7a45d33173a295475f5fea3bc9"
Sep 10 12:43:04.146407 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:04.146375    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:43:09.889640 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:09.889606    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:43:33.966271 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966241    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"]
Sep 10 12:43:33.966743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966676    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy"
Sep 10 12:43:33.966743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966694    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy"
Sep 10 12:43:33.966743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966711    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy"
Sep 10 12:43:33.966743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966718    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy"
Sep 10 12:43:33.966743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966728    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="969fd257-5edf-4ef2-b83a-3d981dddcc19" containerName="payload-pre-processing"
Sep 10 12:43:33.966743 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966738    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="969fd257-5edf-4ef2-b83a-3d981dddcc19" containerName="payload-pre-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966747    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2af580e9-3f13-4203-94fc-40385f2e6611" containerName="main"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966755    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="2af580e9-3f13-4203-94fc-40385f2e6611" containerName="main"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966769    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7f1626bd-1dff-40a7-addd-d6d72ceb800c" containerName="payload-pre-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966777    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="7f1626bd-1dff-40a7-addd-d6d72ceb800c" containerName="payload-pre-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966792    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a510b0fa-6547-41b0-9cfd-27696f31d1d1" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966799    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="a510b0fa-6547-41b0-9cfd-27696f31d1d1" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966811    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966818    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966832    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966840    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966861    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="65eae2b2-17c1-4164-89ed-a7d1d7949505" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966868    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="65eae2b2-17c1-4164-89ed-a7d1d7949505" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966960    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="969fd257-5edf-4ef2-b83a-3d981dddcc19" containerName="payload-pre-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966972    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="65eae2b2-17c1-4164-89ed-a7d1d7949505" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966982    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.966991    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="a510b0fa-6547-41b0-9cfd-27696f31d1d1" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.967000    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="7f1626bd-1dff-40a7-addd-d6d72ceb800c" containerName="payload-pre-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.967007    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.967015    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="bc1b95b2-dc1c-43ff-b49b-1e8b7097f2b9" containerName="payload-processing"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.967023    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.967030    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b3e800d-e06e-4251-8dc7-77573950ade0" containerName="istio-proxy"
Sep 10 12:43:33.967083 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.967037    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="2af580e9-3f13-4203-94fc-40385f2e6611" containerName="main"
Sep 10 12:43:33.970102 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.970085    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:33.972792 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.972776    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-8f6ffb98-kserve-self-signed-certs\""
Sep 10 12:43:33.984806 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:33.984787    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"]
Sep 10 12:43:34.076519 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.076502    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-dshm\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.076621 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.076535    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qthx9\" (UniqueName: \"kubernetes.io/projected/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-kube-api-access-qthx9\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.076621 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.076555    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tmp-dir\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.076621 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.076613    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tls-certs\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.076723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.076651    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-model-cache\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.076723 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.076675    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-home\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.177967 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.177941    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthx9\" (UniqueName: \"kubernetes.io/projected/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-kube-api-access-qthx9\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178082 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.177973    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tmp-dir\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178082 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.177997    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tls-certs\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178082 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.178020    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-model-cache\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178082 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.178047    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-home\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178268 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.178099    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-dshm\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178389 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.178365    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tmp-dir\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178480 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.178385    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-model-cache\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.178533 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.178471    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-home\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.180360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.180342    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-dshm\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.180626 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.180609    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tls-certs\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.186765 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.186744    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthx9\" (UniqueName: \"kubernetes.io/projected/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-kube-api-access-qthx9\") pod \"e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") " pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.278825 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.278773    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:34.557759 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.557690    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"]
Sep 10 12:43:34.561535 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:43:34.561505    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc5e72b38_1e27_4152_a7d4_25e7e78ac1d9.slice/crio-faf2ac5117c0a5fc35f9404ea80fcfd0f51a24f7236d4d73808f35f5bb324769 WatchSource:0}: Error finding container faf2ac5117c0a5fc35f9404ea80fcfd0f51a24f7236d4d73808f35f5bb324769: Status 404 returned error can't find the container with id faf2ac5117c0a5fc35f9404ea80fcfd0f51a24f7236d4d73808f35f5bb324769
Sep 10 12:43:34.718777 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.718740    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw" event={"ID":"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9","Type":"ContainerStarted","Data":"1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d"}
Sep 10 12:43:34.718777 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.718781    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw" event={"ID":"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9","Type":"ContainerStarted","Data":"faf2ac5117c0a5fc35f9404ea80fcfd0f51a24f7236d4d73808f35f5bb324769"}
Sep 10 12:43:34.740755 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:34.740711    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw" podStartSLOduration=1.740697285 podStartE2EDuration="1.740697285s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:43:34.738819396 +0000 UTC m=+2025.935335095" watchObservedRunningTime="2026-09-10 12:43:34.740697285 +0000 UTC m=+2025.937212984"
Sep 10 12:43:36.558153 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:36.558120    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:43:43.971700 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:43.971662    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"]
Sep 10 12:43:43.972142 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:43.971991    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy"
Sep 10 12:43:43.972142 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:43.972003    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="d0e99f80-e8da-4b4d-86b9-af22964a8828" containerName="istio-proxy"
Sep 10 12:43:43.975019 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:43.975003    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:43.977941 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:43.977920    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-8f6ffb98-kserve-self-signed-certs\""
Sep 10 12:43:43.993687 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:43.993665    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"]
Sep 10 12:43:44.049367 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.049344    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tls-certs\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.049491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.049383    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-home\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.049491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.049400    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tmp-dir\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.049491 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.049417    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-dshm\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.049614 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.049507    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-model-cache\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.049614 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.049573    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5kg7\" (UniqueName: \"kubernetes.io/projected/b7905a7b-4338-4b43-b9e4-ce8ed03879db-kube-api-access-t5kg7\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150559 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150534    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-model-cache\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150663 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150578    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5kg7\" (UniqueName: \"kubernetes.io/projected/b7905a7b-4338-4b43-b9e4-ce8ed03879db-kube-api-access-t5kg7\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150663 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150616    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tls-certs\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150663 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150650    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-home\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150663    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tmp-dir\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150810 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150680    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-dshm\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150912 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150871    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-model-cache\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.150994 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.150974    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-home\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.151053 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.151025    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tmp-dir\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.152861 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.152843    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-dshm\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.153116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.153096    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tls-certs\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.159772 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.159751    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5kg7\" (UniqueName: \"kubernetes.io/projected/b7905a7b-4338-4b43-b9e4-ce8ed03879db-kube-api-access-t5kg7\") pod \"e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") " pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.279877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.279830    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:44.279877 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.279874    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:44.284416 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.284393    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:44.292481 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.292461    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:44.571339 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.571315    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"]
Sep 10 12:43:44.573766 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:43:44.573738    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb7905a7b_4338_4b43_b9e4_ce8ed03879db.slice/crio-5577ff2005795f7b6f7e24ee91b968ff1db56a38f574130d903e7d2fab4fe0e5 WatchSource:0}: Error finding container 5577ff2005795f7b6f7e24ee91b968ff1db56a38f574130d903e7d2fab4fe0e5: Status 404 returned error can't find the container with id 5577ff2005795f7b6f7e24ee91b968ff1db56a38f574130d903e7d2fab4fe0e5
Sep 10 12:43:44.758719 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.758680    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt" event={"ID":"b7905a7b-4338-4b43-b9e4-ce8ed03879db","Type":"ContainerStarted","Data":"ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06"}
Sep 10 12:43:44.758719 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.758724    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt" event={"ID":"b7905a7b-4338-4b43-b9e4-ce8ed03879db","Type":"ContainerStarted","Data":"5577ff2005795f7b6f7e24ee91b968ff1db56a38f574130d903e7d2fab4fe0e5"}
Sep 10 12:43:44.769789 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.769764    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:43:44.782154 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:44.782112    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt" podStartSLOduration=1.782098701 podStartE2EDuration="1.782098701s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:43:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:43:44.780030049 +0000 UTC m=+2035.976545749" watchObservedRunningTime="2026-09-10 12:43:44.782098701 +0000 UTC m=+2035.978614399"
Sep 10 12:43:46.651116 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:46.651085    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:43:54.285288 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:54.285258    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:54.285683 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:54.285302    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:54.297555 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:54.297532    2552 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:43:54.811901 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:43:54.811873    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:44:22.502864 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:22.502794    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"]
Sep 10 12:44:24.254066 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:24.254031    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"]
Sep 10 12:44:24.655658 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:24.655624    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:44:32.151903 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:32.151871    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:44:37.521575 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.521516    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw" podUID="c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" containerName="main" containerID="cri-o://1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d" gracePeriod=45
Sep 10 12:44:37.760970 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.760942    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:44:37.848942 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.848884    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:44:37.855468 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855448    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qthx9\" (UniqueName: \"kubernetes.io/projected/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-kube-api-access-qthx9\") pod \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") "
Sep 10 12:44:37.855546 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855483    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tmp-dir\") pod \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") "
Sep 10 12:44:37.855546 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855510    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-dshm\") pod \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") "
Sep 10 12:44:37.855638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855549    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tls-certs\") pod \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") "
Sep 10 12:44:37.855638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855601    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-model-cache\") pod \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") "
Sep 10 12:44:37.855638 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855620    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-home\") pod \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\" (UID: \"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9\") "
Sep 10 12:44:37.855792 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855707    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" (UID: "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:37.855879 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855828    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:37.855940 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855890    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-home" (OuterVolumeSpecName: "home") pod "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" (UID: "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:37.855940 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.855905    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-model-cache" (OuterVolumeSpecName: "model-cache") pod "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" (UID: "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:37.857622 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.857594    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-dshm" (OuterVolumeSpecName: "dshm") pod "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" (UID: "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:37.857730 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.857687    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" (UID: "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:44:37.857730 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.857697    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-kube-api-access-qthx9" (OuterVolumeSpecName: "kube-api-access-qthx9") pod "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" (UID: "c5e72b38-1e27-4152-a7d4-25e7e78ac1d9"). InnerVolumeSpecName "kube-api-access-qthx9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:44:37.957217 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.957197    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qthx9\" (UniqueName: \"kubernetes.io/projected/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-kube-api-access-qthx9\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:37.957217 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.957216    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:37.957341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.957224    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:37.957341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.957233    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:37.957341 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.957240    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:37.965576 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.965550    2552 generic.go:356] "Generic (PLEG): container finished" podID="c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" containerID="1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d" exitCode=0
Sep 10 12:44:37.965661 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.965584    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw" event={"ID":"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9","Type":"ContainerDied","Data":"1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d"}
Sep 10 12:44:37.965661 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.965591    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"
Sep 10 12:44:37.965661 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.965615    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw" event={"ID":"c5e72b38-1e27-4152-a7d4-25e7e78ac1d9","Type":"ContainerDied","Data":"faf2ac5117c0a5fc35f9404ea80fcfd0f51a24f7236d4d73808f35f5bb324769"}
Sep 10 12:44:37.965661 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.965636    2552 scope.go:117] "RemoveContainer" containerID="1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d"
Sep 10 12:44:37.974222 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.974204    2552 scope.go:117] "RemoveContainer" containerID="1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d"
Sep 10 12:44:37.974507 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:44:37.974488    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d\": container with ID starting with 1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d not found: ID does not exist" containerID="1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d"
Sep 10 12:44:37.974581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.974520    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d"} err="failed to get container status \"1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d\": rpc error: code = NotFound desc = could not find container \"1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d\": container with ID starting with 1584b1367e6df9f3fbda52abfd78f80f73b7586ba190d25228678d5e7fefd98d not found: ID does not exist"
Sep 10 12:44:37.995192 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:37.995169    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"]
Sep 10 12:44:38.002416 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:38.002396    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-8f6ffb98-kserve-f4b468787-dv8fw"]
Sep 10 12:44:39.241875 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.241846    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" path="/var/lib/kubelet/pods/c5e72b38-1e27-4152-a7d4-25e7e78ac1d9/volumes"
Sep 10 12:44:39.272992 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.272963    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt" podUID="b7905a7b-4338-4b43-b9e4-ce8ed03879db" containerName="main" containerID="cri-o://ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06" gracePeriod=45
Sep 10 12:44:39.511408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.511386    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:44:39.569182 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569153    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tmp-dir\") pod \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") "
Sep 10 12:44:39.569331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569194    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-dshm\") pod \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") "
Sep 10 12:44:39.569331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569213    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-model-cache\") pod \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") "
Sep 10 12:44:39.569331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569270    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-home\") pod \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") "
Sep 10 12:44:39.569331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569300    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tls-certs\") pod \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") "
Sep 10 12:44:39.569331 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569326    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t5kg7\" (UniqueName: \"kubernetes.io/projected/b7905a7b-4338-4b43-b9e4-ce8ed03879db-kube-api-access-t5kg7\") pod \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\" (UID: \"b7905a7b-4338-4b43-b9e4-ce8ed03879db\") "
Sep 10 12:44:39.569610 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569472    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "b7905a7b-4338-4b43-b9e4-ce8ed03879db" (UID: "b7905a7b-4338-4b43-b9e4-ce8ed03879db"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:39.569610 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569541    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-model-cache" (OuterVolumeSpecName: "model-cache") pod "b7905a7b-4338-4b43-b9e4-ce8ed03879db" (UID: "b7905a7b-4338-4b43-b9e4-ce8ed03879db"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:39.569610 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569572    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-home" (OuterVolumeSpecName: "home") pod "b7905a7b-4338-4b43-b9e4-ce8ed03879db" (UID: "b7905a7b-4338-4b43-b9e4-ce8ed03879db"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:39.569761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569696    2552 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tmp-dir\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:39.569761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569713    2552 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-model-cache\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:39.569761 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.569726    2552 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-home\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:39.571390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.571360    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-dshm" (OuterVolumeSpecName: "dshm") pod "b7905a7b-4338-4b43-b9e4-ce8ed03879db" (UID: "b7905a7b-4338-4b43-b9e4-ce8ed03879db"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 12:44:39.571390 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.571383    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b7905a7b-4338-4b43-b9e4-ce8ed03879db-kube-api-access-t5kg7" (OuterVolumeSpecName: "kube-api-access-t5kg7") pod "b7905a7b-4338-4b43-b9e4-ce8ed03879db" (UID: "b7905a7b-4338-4b43-b9e4-ce8ed03879db"). InnerVolumeSpecName "kube-api-access-t5kg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:44:39.571582 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.571438    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "b7905a7b-4338-4b43-b9e4-ce8ed03879db" (UID: "b7905a7b-4338-4b43-b9e4-ce8ed03879db"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:44:39.670148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.670127    2552 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b7905a7b-4338-4b43-b9e4-ce8ed03879db-tls-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:39.670148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.670148    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5kg7\" (UniqueName: \"kubernetes.io/projected/b7905a7b-4338-4b43-b9e4-ce8ed03879db-kube-api-access-t5kg7\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:39.670273 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.670156    2552 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b7905a7b-4338-4b43-b9e4-ce8ed03879db-dshm\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:39.977308 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.977283    2552 generic.go:356] "Generic (PLEG): container finished" podID="b7905a7b-4338-4b43-b9e4-ce8ed03879db" containerID="ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06" exitCode=0
Sep 10 12:44:39.977482 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.977329    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"
Sep 10 12:44:39.977482 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.977361    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt" event={"ID":"b7905a7b-4338-4b43-b9e4-ce8ed03879db","Type":"ContainerDied","Data":"ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06"}
Sep 10 12:44:39.977482 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.977397    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt" event={"ID":"b7905a7b-4338-4b43-b9e4-ce8ed03879db","Type":"ContainerDied","Data":"5577ff2005795f7b6f7e24ee91b968ff1db56a38f574130d903e7d2fab4fe0e5"}
Sep 10 12:44:39.977482 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.977414    2552 scope.go:117] "RemoveContainer" containerID="ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06"
Sep 10 12:44:39.985454 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.985412    2552 scope.go:117] "RemoveContainer" containerID="ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06"
Sep 10 12:44:39.985703 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:44:39.985686    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06\": container with ID starting with ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06 not found: ID does not exist" containerID="ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06"
Sep 10 12:44:39.985750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.985711    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06"} err="failed to get container status \"ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06\": rpc error: code = NotFound desc = could not find container \"ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06\": container with ID starting with ba454f519088324065073917302e270b73d1499b7b42a48fcd5607c908dcae06 not found: ID does not exist"
Sep 10 12:44:39.998681 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:39.998659    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"]
Sep 10 12:44:40.001554 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:40.001536    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-8f6ffb98-kserve-5558d6c7dc-mbxdt"]
Sep 10 12:44:41.241986 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:41.241955    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b7905a7b-4338-4b43-b9e4-ce8ed03879db" path="/var/lib/kubelet/pods/b7905a7b-4338-4b43-b9e4-ce8ed03879db/volumes"
Sep 10 12:44:44.866616 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:44.866580    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:44:45.280639 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.280553    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"]
Sep 10 12:44:45.280876 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.280864    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" containerName="main"
Sep 10 12:44:45.280920 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.280877    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" containerName="main"
Sep 10 12:44:45.280920 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.280893    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b7905a7b-4338-4b43-b9e4-ce8ed03879db" containerName="main"
Sep 10 12:44:45.280920 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.280898    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="b7905a7b-4338-4b43-b9e4-ce8ed03879db" containerName="main"
Sep 10 12:44:45.281009 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.280947    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="b7905a7b-4338-4b43-b9e4-ce8ed03879db" containerName="main"
Sep 10 12:44:45.281009 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.280957    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5e72b38-1e27-4152-a7d4-25e7e78ac1d9" containerName="main"
Sep 10 12:44:45.285397 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.285382    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.295334 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.290109    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 10 12:44:45.295334 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.290893    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 12:44:45.295334 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.291255    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 10 12:44:45.295334 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.291547    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-89jb9\""
Sep 10 12:44:45.295334 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.291769    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 10 12:44:45.300670 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.300648    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"]
Sep 10 12:44:45.411442 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.411398    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-metrics-certs\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.411549 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.411448    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-maas-api-tls\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.411596 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.411549    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhzf6\" (UniqueName: \"kubernetes.io/projected/aeef94a1-3f8b-436c-a433-8827dd8575d0-kube-api-access-hhzf6\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.411596 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.411579    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeef94a1-3f8b-436c-a433-8827dd8575d0-openshift-service-ca\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.512555 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.512535    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhzf6\" (UniqueName: \"kubernetes.io/projected/aeef94a1-3f8b-436c-a433-8827dd8575d0-kube-api-access-hhzf6\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.512671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.512565    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeef94a1-3f8b-436c-a433-8827dd8575d0-openshift-service-ca\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.512671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.512592    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-metrics-certs\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.512671 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.512610    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-maas-api-tls\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.513178 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.513155    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeef94a1-3f8b-436c-a433-8827dd8575d0-openshift-service-ca\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.514956 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.514937    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-maas-api-tls\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.515051 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.515035    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-metrics-certs\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.521479 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.521461    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhzf6\" (UniqueName: \"kubernetes.io/projected/aeef94a1-3f8b-436c-a433-8827dd8575d0-kube-api-access-hhzf6\") pod \"maas-api-59cc544454-j4xkc\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") " pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.601136 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.601083    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:45.870386 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.870362    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"]
Sep 10 12:44:45.873075 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:44:45.873041    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaeef94a1_3f8b_436c_a433_8827dd8575d0.slice/crio-cd90f816a3d1abb9636707ad232d859923ac36e153fd844d0427c96961d34479 WatchSource:0}: Error finding container cd90f816a3d1abb9636707ad232d859923ac36e153fd844d0427c96961d34479: Status 404 returned error can't find the container with id cd90f816a3d1abb9636707ad232d859923ac36e153fd844d0427c96961d34479
Sep 10 12:44:45.874287 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:45.874269    2552 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 12:44:46.005529 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:46.005499    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc" event={"ID":"aeef94a1-3f8b-436c-a433-8827dd8575d0","Type":"ContainerStarted","Data":"cd90f816a3d1abb9636707ad232d859923ac36e153fd844d0427c96961d34479"}
Sep 10 12:44:48.015267 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:48.015174    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc" event={"ID":"aeef94a1-3f8b-436c-a433-8827dd8575d0","Type":"ContainerStarted","Data":"8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25"}
Sep 10 12:44:48.015654 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:48.015269    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:48.047226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:48.047184    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc" podStartSLOduration=1.155398597 podStartE2EDuration="3.047168768s" totalImagesPullingTime="1.891770171s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:44:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:44:48.04582272 +0000 UTC m=+2099.242338418" watchObservedRunningTime="2026-09-10 12:44:48.047168768 +0000 UTC m=+2099.243684467"
Sep 10 12:44:52.604978 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.604942    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"]
Sep 10 12:44:52.605457 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.605105    2552 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc" podUID="aeef94a1-3f8b-436c-a433-8827dd8575d0" containerName="maas-api" containerID="cri-o://8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25" gracePeriod=30
Sep 10 12:44:52.610485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.610463    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:52.749170 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.749143    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:44:52.838673 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.838650    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:52.968548 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.968528    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeef94a1-3f8b-436c-a433-8827dd8575d0-openshift-service-ca\") pod \"aeef94a1-3f8b-436c-a433-8827dd8575d0\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") "
Sep 10 12:44:52.968701 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.968568    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-maas-api-tls\") pod \"aeef94a1-3f8b-436c-a433-8827dd8575d0\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") "
Sep 10 12:44:52.968701 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.968594    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-metrics-certs\") pod \"aeef94a1-3f8b-436c-a433-8827dd8575d0\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") "
Sep 10 12:44:52.968701 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.968612    2552 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hhzf6\" (UniqueName: \"kubernetes.io/projected/aeef94a1-3f8b-436c-a433-8827dd8575d0-kube-api-access-hhzf6\") pod \"aeef94a1-3f8b-436c-a433-8827dd8575d0\" (UID: \"aeef94a1-3f8b-436c-a433-8827dd8575d0\") "
Sep 10 12:44:52.968922 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.968900    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeef94a1-3f8b-436c-a433-8827dd8575d0-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "aeef94a1-3f8b-436c-a433-8827dd8575d0" (UID: "aeef94a1-3f8b-436c-a433-8827dd8575d0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 12:44:52.970834 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.970812    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "aeef94a1-3f8b-436c-a433-8827dd8575d0" (UID: "aeef94a1-3f8b-436c-a433-8827dd8575d0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:44:52.970917 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.970839    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "aeef94a1-3f8b-436c-a433-8827dd8575d0" (UID: "aeef94a1-3f8b-436c-a433-8827dd8575d0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 12:44:52.970917 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:52.970847    2552 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeef94a1-3f8b-436c-a433-8827dd8575d0-kube-api-access-hhzf6" (OuterVolumeSpecName: "kube-api-access-hhzf6") pod "aeef94a1-3f8b-436c-a433-8827dd8575d0" (UID: "aeef94a1-3f8b-436c-a433-8827dd8575d0"). InnerVolumeSpecName "kube-api-access-hhzf6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 12:44:53.038794 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.038769    2552 generic.go:356] "Generic (PLEG): container finished" podID="aeef94a1-3f8b-436c-a433-8827dd8575d0" containerID="8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25" exitCode=0
Sep 10 12:44:53.038866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.038812    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"
Sep 10 12:44:53.038866 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.038850    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc" event={"ID":"aeef94a1-3f8b-436c-a433-8827dd8575d0","Type":"ContainerDied","Data":"8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25"}
Sep 10 12:44:53.038941 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.038900    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-59cc544454-j4xkc" event={"ID":"aeef94a1-3f8b-436c-a433-8827dd8575d0","Type":"ContainerDied","Data":"cd90f816a3d1abb9636707ad232d859923ac36e153fd844d0427c96961d34479"}
Sep 10 12:44:53.038941 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.038928    2552 scope.go:117] "RemoveContainer" containerID="8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25"
Sep 10 12:44:53.047021 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.047006    2552 scope.go:117] "RemoveContainer" containerID="8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25"
Sep 10 12:44:53.047287 ip-10-0-129-68 kubenswrapper[2552]: E0910 12:44:53.047271    2552 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25\": container with ID starting with 8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25 not found: ID does not exist" containerID="8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25"
Sep 10 12:44:53.047335 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.047295    2552 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25"} err="failed to get container status \"8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25\": rpc error: code = NotFound desc = could not find container \"8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25\": container with ID starting with 8ab4e5250f39c05d9090e99d77b3c5b91792153e0234fa70f6eb665c1f4b5e25 not found: ID does not exist"
Sep 10 12:44:53.069084 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.069061    2552 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"]
Sep 10 12:44:53.069314 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.069298    2552 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-maas-api-tls\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:53.069356 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.069318    2552 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/aeef94a1-3f8b-436c-a433-8827dd8575d0-metrics-certs\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:53.069356 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.069327    2552 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhzf6\" (UniqueName: \"kubernetes.io/projected/aeef94a1-3f8b-436c-a433-8827dd8575d0-kube-api-access-hhzf6\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:53.069356 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.069336    2552 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/aeef94a1-3f8b-436c-a433-8827dd8575d0-openshift-service-ca\") on node \"ip-10-0-129-68.ec2.internal\" DevicePath \"\""
Sep 10 12:44:53.073606 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.073586    2552 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-59cc544454-j4xkc"]
Sep 10 12:44:53.242234 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:53.242186    2552 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeef94a1-3f8b-436c-a433-8827dd8575d0" path="/var/lib/kubelet/pods/aeef94a1-3f8b-436c-a433-8827dd8575d0/volumes"
Sep 10 12:44:59.442281 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:44:59.442252    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:45:00.135210 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.135179    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms"]
Sep 10 12:45:00.135492 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.135479    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeef94a1-3f8b-436c-a433-8827dd8575d0" containerName="maas-api"
Sep 10 12:45:00.135543 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.135493    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeef94a1-3f8b-436c-a433-8827dd8575d0" containerName="maas-api"
Sep 10 12:45:00.135543 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.135542    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeef94a1-3f8b-436c-a433-8827dd8575d0" containerName="maas-api"
Sep 10 12:45:00.139942 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.139927    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms"
Sep 10 12:45:00.142657 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.142635    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-jnqr7\""
Sep 10 12:45:00.147956 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.147932    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms"]
Sep 10 12:45:00.149957 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.149938    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms"
Sep 10 12:45:00.423455 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:00.423371    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms"]
Sep 10 12:45:00.426126 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:45:00.426100    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0451dd29_6d4c_4a27_b93e_4b4af9b6f538.slice/crio-1a4cda3694e44ae7842dd344f3dfb9d6576b1c76c689bb4fa50abfdaa99b22cf WatchSource:0}: Error finding container 1a4cda3694e44ae7842dd344f3dfb9d6576b1c76c689bb4fa50abfdaa99b22cf: Status 404 returned error can't find the container with id 1a4cda3694e44ae7842dd344f3dfb9d6576b1c76c689bb4fa50abfdaa99b22cf
Sep 10 12:45:01.077645 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:01.077611    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms" event={"ID":"0451dd29-6d4c-4a27-b93e-4b4af9b6f538","Type":"ContainerStarted","Data":"d58654c8d27189c8655963751272e330eda24136883e783676e09f2e410d0205"}
Sep 10 12:45:01.077984 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:01.077659    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms" event={"ID":"0451dd29-6d4c-4a27-b93e-4b4af9b6f538","Type":"ContainerStarted","Data":"1a4cda3694e44ae7842dd344f3dfb9d6576b1c76c689bb4fa50abfdaa99b22cf"}
Sep 10 12:45:01.096967 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:01.096839    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms" podStartSLOduration=1.09682262 podStartE2EDuration="1.09682262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:01.095828333 +0000 UTC m=+2112.292344031" watchObservedRunningTime="2026-09-10 12:45:01.09682262 +0000 UTC m=+2112.293338320"
Sep 10 12:45:02.083077 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:02.083044    2552 generic.go:356] "Generic (PLEG): container finished" podID="0451dd29-6d4c-4a27-b93e-4b4af9b6f538" containerID="d58654c8d27189c8655963751272e330eda24136883e783676e09f2e410d0205" exitCode=0
Sep 10 12:45:02.083455 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:02.083125    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms" event={"ID":"0451dd29-6d4c-4a27-b93e-4b4af9b6f538","Type":"ContainerDied","Data":"d58654c8d27189c8655963751272e330eda24136883e783676e09f2e410d0205"}
Sep 10 12:45:03.210287 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:03.210264    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms"
Sep 10 12:45:04.092687 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:04.092664    2552 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms"
Sep 10 12:45:04.092840 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:04.092686    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29817405-fvlms" event={"ID":"0451dd29-6d4c-4a27-b93e-4b4af9b6f538","Type":"ContainerDied","Data":"1a4cda3694e44ae7842dd344f3dfb9d6576b1c76c689bb4fa50abfdaa99b22cf"}
Sep 10 12:45:04.092840 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:04.092728    2552 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1a4cda3694e44ae7842dd344f3dfb9d6576b1c76c689bb4fa50abfdaa99b22cf"
Sep 10 12:45:05.065400 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:05.065364    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:45:11.752271 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:11.752228    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-vf92m"]
Sep 10 12:45:20.373489 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:20.373461    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bl6x9_6b3b51e6-635c-4332-aff7-76fdd9b8e822/manager/0.log"
Sep 10 12:45:21.068644 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:21.068614    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f75798bc7-rqm5n_6a637db7-ef43-4e36-92b1-5b66f960785c/copy-manifests/0.log"
Sep 10 12:45:21.075521 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:21.075499    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f75798bc7-rqm5n_6a637db7-ef43-4e36-92b1-5b66f960785c/manager/0.log"
Sep 10 12:45:21.744846 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:21.744815    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29817405-fvlms_0451dd29-6d4c-4a27-b93e-4b4af9b6f538/cleanup/0.log"
Sep 10 12:45:22.917268 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:22.917239    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xtmsn_4e39e5fa-2a86-4f4e-86c0-059717f4187e/kuadrant-console-plugin/0.log"
Sep 10 12:45:23.149137 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:23.149111    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-vf92m_cfab91b6-0ccc-4e12-bf0b-cc6b9265b791/limitador/0.log"
Sep 10 12:45:23.955796 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:23.955753    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-688cb69f77-h56qn_738e179c-8351-43a8-9f34-d131f8bed321/kube-auth-proxy/0.log"
Sep 10 12:45:25.203684 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:25.203642    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fc8mft_03bdf325-2124-490f-a1a5-4e3bab3ac86c/main/0.log"
Sep 10 12:45:25.420422 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:25.420381    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-m28qc_01e689f2-45b4-4bb9-bc2e-66dd738acae0/main/0.log"
Sep 10 12:45:31.166449 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.166405    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-htg6h/must-gather-ppt8x"]
Sep 10 12:45:31.168744 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.166713    2552 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0451dd29-6d4c-4a27-b93e-4b4af9b6f538" containerName="cleanup"
Sep 10 12:45:31.168744 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.166725    2552 state_mem.go:107] "Deleted CPUSet assignment" podUID="0451dd29-6d4c-4a27-b93e-4b4af9b6f538" containerName="cleanup"
Sep 10 12:45:31.168744 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.166777    2552 memory_manager.go:368] "RemoveStaleState removing state" podUID="0451dd29-6d4c-4a27-b93e-4b4af9b6f538" containerName="cleanup"
Sep 10 12:45:31.169565 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.169550    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.172185 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.172162    2552 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-htg6h\"/\"default-dockercfg-cc8xr\""
Sep 10 12:45:31.172278 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.172166    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-htg6h\"/\"openshift-service-ca.crt\""
Sep 10 12:45:31.172278 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.172219    2552 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-htg6h\"/\"kube-root-ca.crt\""
Sep 10 12:45:31.179274 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.179254    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-htg6h/must-gather-ppt8x"]
Sep 10 12:45:31.253556 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.253529    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d862688-c0ac-4652-9b9c-dc734a3a4f37-must-gather-output\") pod \"must-gather-ppt8x\" (UID: \"2d862688-c0ac-4652-9b9c-dc734a3a4f37\") " pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.253556 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.253557    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szzhm\" (UniqueName: \"kubernetes.io/projected/2d862688-c0ac-4652-9b9c-dc734a3a4f37-kube-api-access-szzhm\") pod \"must-gather-ppt8x\" (UID: \"2d862688-c0ac-4652-9b9c-dc734a3a4f37\") " pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.354416 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.354387    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szzhm\" (UniqueName: \"kubernetes.io/projected/2d862688-c0ac-4652-9b9c-dc734a3a4f37-kube-api-access-szzhm\") pod \"must-gather-ppt8x\" (UID: \"2d862688-c0ac-4652-9b9c-dc734a3a4f37\") " pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.354529 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.354477    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d862688-c0ac-4652-9b9c-dc734a3a4f37-must-gather-output\") pod \"must-gather-ppt8x\" (UID: \"2d862688-c0ac-4652-9b9c-dc734a3a4f37\") " pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.354733 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.354719    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/2d862688-c0ac-4652-9b9c-dc734a3a4f37-must-gather-output\") pod \"must-gather-ppt8x\" (UID: \"2d862688-c0ac-4652-9b9c-dc734a3a4f37\") " pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.362915 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.362896    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szzhm\" (UniqueName: \"kubernetes.io/projected/2d862688-c0ac-4652-9b9c-dc734a3a4f37-kube-api-access-szzhm\") pod \"must-gather-ppt8x\" (UID: \"2d862688-c0ac-4652-9b9c-dc734a3a4f37\") " pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.478664 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.478610    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-htg6h/must-gather-ppt8x"
Sep 10 12:45:31.754878 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:31.754741    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-htg6h/must-gather-ppt8x"]
Sep 10 12:45:31.757607 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:45:31.757579    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2d862688_c0ac_4652_9b9c_dc734a3a4f37.slice/crio-5d23f1ddced988807a67a752db2508d930a0b3477f90b32770833fbeca1878ac WatchSource:0}: Error finding container 5d23f1ddced988807a67a752db2508d930a0b3477f90b32770833fbeca1878ac: Status 404 returned error can't find the container with id 5d23f1ddced988807a67a752db2508d930a0b3477f90b32770833fbeca1878ac
Sep 10 12:45:32.212369 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:32.212335    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-htg6h/must-gather-ppt8x" event={"ID":"2d862688-c0ac-4652-9b9c-dc734a3a4f37","Type":"ContainerStarted","Data":"5d23f1ddced988807a67a752db2508d930a0b3477f90b32770833fbeca1878ac"}
Sep 10 12:45:33.219647 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:33.219504    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-htg6h/must-gather-ppt8x" event={"ID":"2d862688-c0ac-4652-9b9c-dc734a3a4f37","Type":"ContainerStarted","Data":"89566db7a9215d9e7d1b1f7185497c952bbf1a1517f4ae0e333d3489d4a4f6bc"}
Sep 10 12:45:33.219647 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:33.219559    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-htg6h/must-gather-ppt8x" event={"ID":"2d862688-c0ac-4652-9b9c-dc734a3a4f37","Type":"ContainerStarted","Data":"22d706584ce6ca25cdc2c6ea00cb83d46001aa909570faefaf396aad1fb0ee99"}
Sep 10 12:45:34.209872 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:34.209841    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w7g5k_c982e9cd-3e88-4e3e-9d59-4bbca5053412/global-pull-secret-syncer/0.log"
Sep 10 12:45:34.395420 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:34.395385    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-gdnqz_e96c83bb-d317-452c-8326-0e996ab29ebf/konnectivity-agent/0.log"
Sep 10 12:45:34.464350 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:34.464269    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-68.ec2.internal_72d08dde1ef99723fcb5d6e25dbc85b3/haproxy/0.log"
Sep 10 12:45:38.947698 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:38.947670    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-xtmsn_4e39e5fa-2a86-4f4e-86c0-059717f4187e/kuadrant-console-plugin/0.log"
Sep 10 12:45:39.032408 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:39.032383    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-vf92m_cfab91b6-0ccc-4e12-bf0b-cc6b9265b791/limitador/0.log"
Sep 10 12:45:41.165675 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.165645    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8l75_adf510c5-b4ea-49ff-8724-edf51e127bef/node-exporter/0.log"
Sep 10 12:45:41.190381 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.190273    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8l75_adf510c5-b4ea-49ff-8724-edf51e127bef/kube-rbac-proxy/0.log"
Sep 10 12:45:41.226520 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.224358    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-z8l75_adf510c5-b4ea-49ff-8724-edf51e127bef/init-textfile/0.log"
Sep 10 12:45:41.258820 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.258602    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-v8wsv_2e08cbc4-613b-4638-85be-84fbd1e335b8/kube-rbac-proxy-main/0.log"
Sep 10 12:45:41.281485 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.281454    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-v8wsv_2e08cbc4-613b-4638-85be-84fbd1e335b8/kube-rbac-proxy-self/0.log"
Sep 10 12:45:41.309800 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.309773    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-657fd6c54-v8wsv_2e08cbc4-613b-4638-85be-84fbd1e335b8/openshift-state-metrics/0.log"
Sep 10 12:45:41.352112 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.352089    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_987b7840-2c80-4683-a8d4-fff29d1c838d/prometheus/0.log"
Sep 10 12:45:41.380652 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.380626    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_987b7840-2c80-4683-a8d4-fff29d1c838d/config-reloader/0.log"
Sep 10 12:45:41.408226 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.408185    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_987b7840-2c80-4683-a8d4-fff29d1c838d/thanos-sidecar/0.log"
Sep 10 12:45:41.434581 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.434557    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_987b7840-2c80-4683-a8d4-fff29d1c838d/kube-rbac-proxy-web/0.log"
Sep 10 12:45:41.467211 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.467187    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_987b7840-2c80-4683-a8d4-fff29d1c838d/kube-rbac-proxy/0.log"
Sep 10 12:45:41.498415 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.498388    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_987b7840-2c80-4683-a8d4-fff29d1c838d/kube-rbac-proxy-thanos/0.log"
Sep 10 12:45:41.532113 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.532041    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_987b7840-2c80-4683-a8d4-fff29d1c838d/init-config-reloader/0.log"
Sep 10 12:45:41.710360 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.710327    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8f75f756c-q9g5f_178e1b02-25a5-4199-bb03-40324606309c/telemeter-client/0.log"
Sep 10 12:45:41.767306 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.767272    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8f75f756c-q9g5f_178e1b02-25a5-4199-bb03-40324606309c/reload/0.log"
Sep 10 12:45:41.817398 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:41.817301    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-8f75f756c-q9g5f_178e1b02-25a5-4199-bb03-40324606309c/kube-rbac-proxy/0.log"
Sep 10 12:45:43.703460 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.702501    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-htg6h/must-gather-ppt8x" podStartSLOduration=11.880200319 podStartE2EDuration="12.702474822s" totalImagesPullingTime="822.274503ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 12:45:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:33.241383714 +0000 UTC m=+2144.437899414" watchObservedRunningTime="2026-09-10 12:45:43.702474822 +0000 UTC m=+2154.898990524"
Sep 10 12:45:43.704039 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.703495    2552 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"]
Sep 10 12:45:43.709857 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.709829    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.722507 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.722482    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"]
Sep 10 12:45:43.774342 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.774302    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-lib-modules\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.774536 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.774358    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rjw7x\" (UniqueName: \"kubernetes.io/projected/837c8442-b42f-4b8d-9c2d-d9a437972115-kube-api-access-rjw7x\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.774536 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.774387    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-proc\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.774536 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.774438    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-podres\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.774536 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.774508    2552 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-sys\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875184 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875153    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-sys\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875366 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875216    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-lib-modules\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875366 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875246    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rjw7x\" (UniqueName: \"kubernetes.io/projected/837c8442-b42f-4b8d-9c2d-d9a437972115-kube-api-access-rjw7x\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875366 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875265    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-proc\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875366 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875290    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-sys\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875366 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875304    2552 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-podres\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875390    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-proc\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875447    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-lib-modules\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.875591 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.875485    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/837c8442-b42f-4b8d-9c2d-d9a437972115-podres\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:43.885312 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:43.885275    2552 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rjw7x\" (UniqueName: \"kubernetes.io/projected/837c8442-b42f-4b8d-9c2d-d9a437972115-kube-api-access-rjw7x\") pod \"perf-node-gather-daemonset-9lnc8\" (UID: \"837c8442-b42f-4b8d-9c2d-d9a437972115\") " pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:44.020012 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:44.019917    2552 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:44.308441 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:44.308391    2552 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"]
Sep 10 12:45:44.311056 ip-10-0-129-68 kubenswrapper[2552]: W0910 12:45:44.311031    2552 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod837c8442_b42f_4b8d_9c2d_d9a437972115.slice/crio-d867b17cf1eb3a45c55a646e22da972dfe438d79e8679465e0579ce4d047640a WatchSource:0}: Error finding container d867b17cf1eb3a45c55a646e22da972dfe438d79e8679465e0579ce4d047640a: Status 404 returned error can't find the container with id d867b17cf1eb3a45c55a646e22da972dfe438d79e8679465e0579ce4d047640a
Sep 10 12:45:45.316148 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:45.316105    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8" event={"ID":"837c8442-b42f-4b8d-9c2d-d9a437972115","Type":"ContainerStarted","Data":"19ce38cc8c0f6aae62f7354560aaf1f287174198ccca197ca0dba22d0255025b"}
Sep 10 12:45:45.316678 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:45.316175    2552 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8" event={"ID":"837c8442-b42f-4b8d-9c2d-d9a437972115","Type":"ContainerStarted","Data":"d867b17cf1eb3a45c55a646e22da972dfe438d79e8679465e0579ce4d047640a"}
Sep 10 12:45:45.316678 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:45.316216    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:45.335220 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:45.335165    2552 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8" podStartSLOduration=2.335143872 podStartE2EDuration="2.335143872s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 12:45:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 12:45:45.333381023 +0000 UTC m=+2156.529896723" watchObservedRunningTime="2026-09-10 12:45:45.335143872 +0000 UTC m=+2156.531659573"
Sep 10 12:45:46.472255 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:46.472229    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxvfc_9e7f3c5e-3819-44f9-b6ff-94f466858b2c/dns/0.log"
Sep 10 12:45:46.508689 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:46.508661    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-xxvfc_9e7f3c5e-3819-44f9-b6ff-94f466858b2c/kube-rbac-proxy/0.log"
Sep 10 12:45:46.609565 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:46.609539    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-rfpvc_bbfe1596-ceaa-4f73-aa21-59bb62bf1527/dns-node-resolver/0.log"
Sep 10 12:45:47.148843 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:47.148812    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-dffdcb8ff-rfpg2_b9e1f7f5-f5af-426d-a702-d2989ae3c6a1/registry/0.log"
Sep 10 12:45:47.215750 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:47.215713    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-f8nkn_461c173a-3009-4c70-a4c1-ba0ca3e8b945/node-ca/0.log"
Sep 10 12:45:48.527061 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:48.527033    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-688cb69f77-h56qn_738e179c-8351-43a8-9f34-d131f8bed321/kube-auth-proxy/0.log"
Sep 10 12:45:49.608102 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:49.608074    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-cmhtc_02f33ccf-a68b-4de0-822c-5e710ed402db/serve-healthcheck-canary/0.log"
Sep 10 12:45:50.291114 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:50.291086    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77b6m_c268d932-b3fa-4d93-a76d-3fba80f645e2/kube-rbac-proxy/0.log"
Sep 10 12:45:50.316227 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:50.316198    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77b6m_c268d932-b3fa-4d93-a76d-3fba80f645e2/exporter/0.log"
Sep 10 12:45:50.342117 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:50.342095    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-77b6m_c268d932-b3fa-4d93-a76d-3fba80f645e2/extractor/0.log"
Sep 10 12:45:51.331076 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:51.331049    2552 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-htg6h/perf-node-gather-daemonset-9lnc8"
Sep 10 12:45:53.038138 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:53.038108    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-bl6x9_6b3b51e6-635c-4332-aff7-76fdd9b8e822/manager/0.log"
Sep 10 12:45:53.444049 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:53.444019    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f75798bc7-rqm5n_6a637db7-ef43-4e36-92b1-5b66f960785c/manager/0.log"
Sep 10 12:45:53.469955 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:53.469924    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-5f75798bc7-rqm5n_6a637db7-ef43-4e36-92b1-5b66f960785c/copy-manifests/0.log"
Sep 10 12:45:54.105387 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:54.105308    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-688cb69f77-h56qn_738e179c-8351-43a8-9f34-d131f8bed321/kube-auth-proxy/0.log"
Sep 10 12:45:54.731163 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:54.731125    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-74996bc6f6-rs762_6841d4a2-2f4c-4cb6-b420-6715cc0170be/manager/0.log"
Sep 10 12:45:55.955389 ip-10-0-129-68 kubenswrapper[2552]: I0910 12:45:55.955346    2552 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-688cb69f77-h56qn_738e179c-8351-43a8-9f34-d131f8bed321/kube-auth-proxy/0.log"